Nov 04 01:07:31.612947 ip-10-0-138-179 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 04 01:07:31.612957 ip-10-0-138-179 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 04 01:07:31.612964 ip-10-0-138-179 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 04 01:07:31.613177 ip-10-0-138-179 systemd[1]: Failed to start Kubernetes Kubelet. Nov 04 01:07:41.669755 ip-10-0-138-179 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 04 01:07:41.669770 ip-10-0-138-179 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot c21fe9391ab84b028f68f34af617cb1f -- Nov 04 01:09:35.194668 ip-10-0-138-179 systemd[1]: Starting Kubernetes Kubelet... Nov 04 01:09:35.657098 ip-10-0-138-179 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 01:09:35.657098 ip-10-0-138-179 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 01:09:35.657098 ip-10-0-138-179 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 01:09:35.657098 ip-10-0-138-179 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 01:09:35.663192 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663118 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:09:35.663192 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663189 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:09:35.663192 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663194 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:09:35.663192 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663197 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:09:35.663192 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663199 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663202 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663205 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663208 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663210 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663213 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663215 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663218 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663220 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663223 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663226 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663228 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663231 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663233 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663236 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663238 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663241 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663243 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663246 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663248 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:09:35.663381 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663251 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663253 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663255 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663258 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663261 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663263 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663265 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663268 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663271 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663274 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663276 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663279 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663282 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663284 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663287 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663290 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663292 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663295 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663297 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663300 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:09:35.663849 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663302 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663305 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663307 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663310 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663312 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663315 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663317 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663320 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663323 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663327 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663331 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663334 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663336 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663339 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663342 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663345 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663347 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663350 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663354 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:09:35.664294 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663356 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663359 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663362 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663365 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663367 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663370 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663373 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663377 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663381 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663383 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663386 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663389 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663392 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663395 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663397 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663400 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663403 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663405 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663407 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:09:35.664741 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663410 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663427 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663430 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663432 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663435 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663438 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663440 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663443 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663446 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663448 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663451 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663453 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663456 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663460 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663463 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663466 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663469 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663471 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663474 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:09:35.665232 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663476 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663479 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663481 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663484 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.663487 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664251 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664257 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664260 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664263 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664266 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664268 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664271 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664274 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664276 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664279 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664282 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664284 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664287 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664289 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664292 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:09:35.665731 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664295 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664297 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664300 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664304 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664308 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664311 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664314 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664317 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664320 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664323 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664325 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664328 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664330 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664333 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664335 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664338 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664341 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664343 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664346 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:09:35.666213 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664349 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664351 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664354 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664356 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664358 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664362 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664366 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664369 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664372 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664375 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664378 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664380 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664382 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664385 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664388 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664390 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664392 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664395 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664398 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:09:35.666674 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664400 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664403 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664405 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664408 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664411 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664429 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664431 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664435 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664438 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664440 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664443 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664445 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664448 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664450 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664453 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664455 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664458 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664460 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664463 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664465 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:09:35.667121 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664467 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664470 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664472 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664476 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664479 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664481 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664484 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664487 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664489 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664491 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664494 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664497 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664499 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664501 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664504 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664507 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664510 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664512 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664515 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664517 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:09:35.667618 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664520 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664523 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664525 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664528 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664530 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664533 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664535 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664538 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664540 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664543 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664545 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664548 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.664551 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665040 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665046 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665049 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665052 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665055 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665058 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:09:35.668079 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665060 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665063 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665066 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665069 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665071 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665073 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665076 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665079 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665081 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665084 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665087 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665089 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665092 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665096 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665100 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665103 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665109 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665112 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665115 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:09:35.668546 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665117 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665120 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665122 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665125 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665128 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665130 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665132 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665135 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665138 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665141 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665144 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665147 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665150 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665153 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665156 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665158 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665161 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665163 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665166 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665169 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:09:35.669005 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665171 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665174 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665176 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665179 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665181 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665184 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665187 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665189 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665192 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665195 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665197 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665200 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665202 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665205 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665207 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665210 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665213 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665215 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665218 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:09:35.669489 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665220 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665223 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665225 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665228 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665232 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665235 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665238 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665241 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665243 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665246 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665248 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665251 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665254 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665257 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665259 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665262 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665264 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665267 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665270 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665273 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:09:35.669993 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665275 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665278 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665281 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665283 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665285 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665289 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665293 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665296 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665298 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665301 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665303 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665306 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665308 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665311 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665314 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665316 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665319 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665323 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665326 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665328 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:09:35.670508 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665331 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.665333 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665451 2580 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665461 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665473 2580 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665479 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665484 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665487 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665492 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665496 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665500 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665503 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665506 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665510 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665513 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665516 2580 flags.go:64] FLAG: --cgroup-root="" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665519 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665522 2580 flags.go:64] FLAG: --client-ca-file="" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665525 2580 flags.go:64] FLAG: --cloud-provider="external" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665529 2580 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665535 2580 flags.go:64] FLAG: --cluster-domain="" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665538 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665541 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665544 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 01:09:35.670973 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665551 2580 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665557 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665560 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665563 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665566 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665569 2580 flags.go:64] FLAG: --contention-profiling="false" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665573 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665576 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665580 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665583 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665587 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665590 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665593 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665596 2580 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665599 2580 flags.go:64] FLAG: --enable-server="true" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665602 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665606 2580 flags.go:64] FLAG: --event-burst="100" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665609 2580 flags.go:64] FLAG: --event-qps="50" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665612 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665615 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665618 2580 flags.go:64] FLAG: --eviction-hard="" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665623 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665626 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665629 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665632 2580 flags.go:64] FLAG: --eviction-soft="" Nov 04 01:09:35.671561 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665635 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665638 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665641 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665644 2580 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665647 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665650 2580 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665653 2580 flags.go:64] FLAG: --feature-gates="" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665661 2580 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665666 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665669 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665672 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665675 2580 flags.go:64] FLAG: --healthz-port="10248" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665679 2580 flags.go:64] FLAG: --help="false" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665682 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665685 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665688 2580 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665691 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665695 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665698 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665701 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665704 2580 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665707 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665710 2580 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 01:09:35.672160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665713 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665716 2580 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665719 2580 flags.go:64] FLAG: --kube-reserved="" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665722 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665725 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665728 2580 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665731 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665734 2580 flags.go:64] FLAG: --lock-file="" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665737 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665740 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665744 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665749 2580 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665752 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665755 2580 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665758 2580 flags.go:64] FLAG: --logging-format="text" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665761 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665765 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665770 2580 flags.go:64] FLAG: --manifest-url="" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665773 2580 flags.go:64] FLAG: --manifest-url-header="" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665777 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665781 2580 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665785 2580 flags.go:64] FLAG: --max-pods="110" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665788 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665792 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665795 2580 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 01:09:35.672721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665798 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665801 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665804 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665807 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665814 2580 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665817 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665821 2580 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665824 2580 flags.go:64] FLAG: --pod-cidr="" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665827 2580 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665830 2580 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665833 2580 flags.go:64] FLAG: --pods-per-core="0" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665836 2580 flags.go:64] FLAG: --port="10250" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665839 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665842 2580 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-09334c893b020cf8c" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665845 2580 flags.go:64] FLAG: --qos-reserved="" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665848 2580 flags.go:64] FLAG: --read-only-port="10255" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665851 2580 flags.go:64] FLAG: --register-node="true" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665854 2580 flags.go:64] FLAG: --register-with-taints="" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665858 2580 flags.go:64] FLAG: --registry-burst="10" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665861 2580 flags.go:64] FLAG: --registry-qps="5" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665864 2580 flags.go:64] FLAG: --reserved-cpus="" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665867 2580 flags.go:64] FLAG: --reserved-memory="" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665870 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665874 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665879 2580 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665882 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 01:09:35.673315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665885 2580 flags.go:64] FLAG: --runonce="false" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665888 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665891 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665894 2580 flags.go:64] FLAG: --seccomp-default="false" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665897 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665900 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665908 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665911 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665914 2580 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665917 2580 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665920 2580 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665923 2580 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665926 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665929 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665932 2580 flags.go:64] FLAG: --system-cgroups="" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665935 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665940 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665943 2580 flags.go:64] FLAG: --tls-cert-file="" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665946 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665950 2580 flags.go:64] FLAG: --tls-min-version="" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665953 2580 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665956 2580 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665959 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665962 2580 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665965 2580 flags.go:64] FLAG: --v="2" Nov 04 01:09:35.673980 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665969 2580 flags.go:64] FLAG: --version="false" Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665974 2580 flags.go:64] FLAG: --vmodule="" Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665978 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.665981 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666117 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666122 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666125 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666128 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666131 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666133 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666136 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666138 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666141 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666145 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666150 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666153 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666157 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666160 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666162 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666165 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:09:35.674562 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666168 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666170 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666173 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666175 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666178 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666181 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666183 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666186 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666188 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666191 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666193 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666196 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666198 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666201 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666203 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666206 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666210 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666214 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666218 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:09:35.675039 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666220 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666224 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666226 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666229 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666232 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666234 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666237 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666239 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666243 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666245 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666248 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666251 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666256 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666259 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666261 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666265 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666267 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666270 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666272 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:09:35.675561 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666275 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666277 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666280 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666283 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666285 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666288 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666290 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666293 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666295 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666298 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666301 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666305 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666308 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666310 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666313 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666315 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666318 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666321 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666323 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666326 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:09:35.676009 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666328 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666331 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666334 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666337 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666339 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666342 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666344 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666347 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666349 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666352 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666355 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666357 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666360 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666362 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666365 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666367 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666370 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666372 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666375 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666378 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:09:35.676480 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666380 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:09:35.676945 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666383 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:09:35.676945 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666386 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:09:35.676945 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666390 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:09:35.676945 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666393 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:09:35.676945 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666396 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:09:35.676945 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666398 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:09:35.676945 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666401 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:09:35.676945 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666404 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:09:35.676945 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666406 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:09:35.676945 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666409 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:09:35.676945 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666423 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:09:35.676945 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.666449 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 01:09:35.676945 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666487 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:09:35.676945 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666492 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:09:35.676945 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666496 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:09:35.676945 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666499 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666502 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666505 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666508 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666510 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666513 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666516 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666518 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666521 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666523 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666526 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666529 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666531 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666534 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666537 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666539 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666542 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666545 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666549 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666552 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:09:35.677329 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666556 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666559 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666561 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666564 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666567 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666570 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666572 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666575 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666577 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666580 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666583 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666585 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666588 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666591 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666593 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666596 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666598 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666601 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666604 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666606 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:09:35.677822 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666609 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666612 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666616 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666618 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666621 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666624 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666627 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666629 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666632 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666635 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666638 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666641 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666645 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666647 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666650 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666652 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666655 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666658 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666660 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:09:35.678290 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666663 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666665 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666668 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666671 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666673 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666676 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666679 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666683 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666685 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666688 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666691 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666694 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666696 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666699 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666701 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666704 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666706 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666709 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666712 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:09:35.678755 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666716 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666719 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666722 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666725 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666729 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666731 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666735 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666738 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666741 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666743 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666745 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666748 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666751 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666753 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666756 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666758 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666761 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666763 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666766 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666769 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:09:35.679225 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666772 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666774 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666777 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666779 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.666783 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.673829 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.673845 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.673858 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.673863 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673922 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673928 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673931 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673934 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673937 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673940 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673942 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673945 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673948 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673950 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673953 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:09:35.679762 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673956 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673958 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673961 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673964 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673967 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673969 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673972 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673975 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673978 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673980 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673983 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673986 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673988 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673991 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673994 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673997 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.673999 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674002 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674006 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674009 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:09:35.680219 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674012 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674014 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674017 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674019 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674022 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674025 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674027 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674030 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674032 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674035 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674038 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674040 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674043 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674045 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674049 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674054 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674057 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674060 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674062 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:09:35.680693 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674065 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674067 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674070 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674073 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674076 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674078 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674081 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674084 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674087 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674090 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674093 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674096 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674099 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674102 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674105 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674107 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674110 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674113 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674115 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:09:35.681200 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674118 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674121 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674123 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674126 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674128 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674131 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674134 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674136 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674139 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674141 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674144 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674146 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674150 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674155 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674158 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674161 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674164 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674166 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674169 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:09:35.681956 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674172 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:09:35.682541 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674174 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:09:35.682541 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674177 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:09:35.682541 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674179 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:09:35.682541 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674183 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:09:35.682541 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674186 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:09:35.682541 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674189 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:09:35.682541 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674192 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:09:35.682541 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674194 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:09:35.682541 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674197 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:09:35.682541 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674199 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:09:35.682541 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674202 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:09:35.682541 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674205 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:09:35.682541 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674207 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:09:35.682541 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674210 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:09:35.682541 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674212 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:09:35.682541 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674215 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:09:35.682541 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674217 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:09:35.682941 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.674246 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 01:09:35.682941 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674396 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:09:35.682941 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674401 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:09:35.682941 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674403 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:09:35.682941 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674406 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:09:35.682941 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674409 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:09:35.682941 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674426 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:09:35.682941 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674429 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:09:35.682941 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674432 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:09:35.682941 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674435 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:09:35.682941 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674438 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:09:35.682941 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674440 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:09:35.682941 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674444 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:09:35.682941 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674446 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:09:35.682941 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674449 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:09:35.682941 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674452 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674455 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674457 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674460 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674463 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674467 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674470 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674473 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674476 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674478 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674481 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674483 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674486 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674489 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674492 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674494 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674497 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674499 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674502 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:09:35.683321 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674505 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674507 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674510 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674512 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674515 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674517 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674520 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674522 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674525 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674529 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674531 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674534 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674536 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674539 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674541 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674544 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674546 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674549 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674552 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674554 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:09:35.683793 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674557 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674560 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674563 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674567 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674570 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674574 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674576 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674579 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674582 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674584 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674587 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674589 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674592 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674594 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674597 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674599 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674602 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674605 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674607 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:09:35.684289 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674610 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674613 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674615 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674618 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674621 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674624 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674626 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674630 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674634 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674637 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674640 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674643 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674645 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674648 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674651 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674654 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674656 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674659 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674661 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:09:35.684784 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674664 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:09:35.685237 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674666 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:09:35.685237 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674669 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:09:35.685237 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674671 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:09:35.685237 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674674 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:09:35.685237 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674676 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:09:35.685237 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674679 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:09:35.685237 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674682 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:09:35.685237 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674684 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:09:35.685237 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674687 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:09:35.685237 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674689 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:09:35.685237 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674692 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:09:35.685237 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674695 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:09:35.685237 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674697 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:09:35.685237 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674700 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:09:35.685237 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.674723 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674754 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674759 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674763 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674766 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674768 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674771 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674774 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674777 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674779 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674782 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674784 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674787 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674789 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674793 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674795 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674798 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674800 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674803 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:09:35.685660 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674805 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674808 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674810 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674812 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674815 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674817 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674820 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674823 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674825 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674828 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674832 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674835 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674837 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674840 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674842 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674845 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674847 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674850 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674852 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:09:35.686171 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674855 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674858 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674860 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674863 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674865 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674868 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674870 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674873 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674876 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674879 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674881 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674884 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674886 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674889 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674891 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674893 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674896 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674898 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674901 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:09:35.686712 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674903 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674906 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674908 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674911 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674915 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674917 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674920 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674922 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674925 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674927 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674930 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674932 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674935 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674937 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674940 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674943 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674945 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674948 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674950 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:09:35.687177 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674953 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674955 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674957 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674960 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674963 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674965 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674968 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674971 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674975 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674978 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674981 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674983 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674987 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674990 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674993 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674996 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.674998 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.675002 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.675005 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:09:35.687689 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.675008 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:09:35.688175 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.675010 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:09:35.688175 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.675013 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:09:35.688175 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.675015 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:09:35.688175 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.675018 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:09:35.688175 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.675021 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:09:35.688175 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.675023 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:09:35.688175 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.675025 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:09:35.688175 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.675028 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:09:35.688175 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.675030 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:09:35.688175 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.675034 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:09:35.688175 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:35.675036 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:09:35.688175 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.675086 2580 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 01:09:35.688175 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.677196 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 04 01:09:35.688175 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.678184 2580 server.go:1014] "Starting client certificate rotation" Nov 04 01:09:35.688175 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.678280 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 01:09:35.688175 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.679039 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 01:09:35.688175 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.680551 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 01:09:35.688175 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.685703 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 01:09:35.700764 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.700748 2580 log.go:25] "Validated CRI v1 runtime API" Nov 04 01:09:35.704279 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.704264 2580 log.go:25] "Validated CRI v1 image API" Nov 04 01:09:35.705123 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.705103 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 01:09:35.709895 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.709877 2580 fs.go:135] Filesystem UUIDs: map[735397ad-cda3-45de-883c-dbbcfb80bb89:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 972bd2db-728e-44e9-9585-9de78c2caa26:/dev/nvme0n1p4] Nov 04 01:09:35.709942 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.709895 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 04 01:09:35.716256 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.716150 2580 manager.go:217] Machine: {Timestamp:2026-11-04 01:09:35.714612898 +0000 UTC m=+0.391040536 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3164237 MemoryCapacity:33164345344 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e59fb50382bd07caaa74aa1274482 SystemUUID:ec2e59fb-5038-2bd0-7caa-a74aa1274482 BootID:c21fe939-1ab8-4b02-8f68-f34af617cb1f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582172672 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:da:1b:da:27 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:da:1b:da:27 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:7c:36:62:5b:92 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164345344 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 01:09:35.716256 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.716250 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 01:09:35.716368 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.716347 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 01:09:35.719787 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.719766 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 01:09:35.719909 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.719788 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-179.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 01:09:35.720676 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.720666 2580 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 01:09:35.720712 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.720677 2580 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 01:09:35.720712 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.720687 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 01:09:35.720712 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.720697 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 01:09:35.721829 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.721819 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 01:09:35.726068 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.726045 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 01:09:35.726208 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.726199 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 01:09:35.726949 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.726940 2580 kubelet.go:493] "Attempting to sync node with API server" Nov 04 01:09:35.726979 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.726956 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 01:09:35.726979 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.726966 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 01:09:35.726979 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.726974 2580 kubelet.go:405] "Adding apiserver pod source" Nov 04 01:09:35.727074 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.726982 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 01:09:35.727271 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.727259 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 01:09:35.730013 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.729987 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 01:09:35.737636 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.737617 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 01:09:35.738805 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.738792 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 01:09:35.738856 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.738822 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 01:09:35.738986 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.738976 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 01:09:35.739023 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.738991 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 01:09:35.739023 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.738998 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 01:09:35.739023 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.739003 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 01:09:35.739023 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.739008 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 01:09:35.739023 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.739013 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 01:09:35.739608 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.739599 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 01:09:35.739645 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.739609 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 01:09:35.739645 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.739617 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 01:09:35.739645 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.739623 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 01:09:35.739645 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.739631 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 01:09:35.739645 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.739640 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 01:09:35.740794 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.740784 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 01:09:35.740794 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.740794 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 01:09:35.744453 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.744441 2580 server.go:1263] "Started kubelet" Nov 04 01:09:35.744633 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.744594 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 01:09:35.744794 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.744586 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 01:09:35.744878 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.744825 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 01:09:35.745060 ip-10-0-138-179 systemd[1]: Started Kubernetes Kubelet. Nov 04 01:09:35.746310 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.746292 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 01:09:35.747182 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.747168 2580 server.go:317] "Adding debug handlers to kubelet server" Nov 04 01:09:35.750765 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:35.750747 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 04 01:09:35.752325 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.752310 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 01:09:35.752325 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.752320 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 01:09:35.752480 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:35.752438 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-179.us-west-2.compute.internal\" not found" Nov 04 01:09:35.753034 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.753016 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 01:09:35.753034 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.753018 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 01:09:35.753179 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.753042 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 01:09:35.753179 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.753138 2580 reconstruct.go:97] "Volume reconstruction finished" Nov 04 01:09:35.753179 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.753146 2580 reconciler.go:29] "Reconciler: start to sync state" Nov 04 01:09:35.753542 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.753525 2580 factory.go:55] Registering systemd factory Nov 04 01:09:35.753608 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.753546 2580 factory.go:223] Registration of the systemd container factory successfully Nov 04 01:09:35.753794 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.753780 2580 factory.go:153] Registering CRI-O factory Nov 04 01:09:35.753849 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.753796 2580 factory.go:223] Registration of the crio container factory successfully Nov 04 01:09:35.753849 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.753840 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 01:09:35.753986 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.753859 2580 factory.go:103] Registering Raw factory Nov 04 01:09:35.753986 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.753875 2580 manager.go:1199] Started watching for new ooms in manager Nov 04 01:09:35.754948 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.754932 2580 manager.go:319] Starting recovery of all containers Nov 04 01:09:35.765584 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.765560 2580 manager.go:324] Recovery completed Nov 04 01:09:35.769982 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.769965 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 01:09:35.771735 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.771713 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 01:09:35.771798 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.771747 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 01:09:35.771798 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.771757 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 01:09:35.772208 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.772194 2580 cpu_manager.go:226] "Starting" policy="none" Nov 04 01:09:35.772208 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.772207 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 01:09:35.772325 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.772225 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 01:09:35.772325 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.772199 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 01:09:35.772932 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.772919 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 01:09:35.773009 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.772940 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 01:09:35.773009 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.772951 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 01:09:35.774692 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.774679 2580 policy_none.go:50] "Start" Nov 04 01:09:35.774692 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.774695 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 01:09:35.774786 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.774706 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 01:09:35.776684 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.776674 2580 policy_none.go:44] "Start" Nov 04 01:09:35.828516 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.807942 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 01:09:35.828516 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.810956 2580 manager.go:341] "Starting Device Plugin manager" Nov 04 01:09:35.828516 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:35.810999 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 01:09:35.828516 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.811011 2580 server.go:87] "Starting device plugin registration server" Nov 04 01:09:35.828516 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.811238 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 01:09:35.828516 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.811304 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 01:09:35.828516 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.811404 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 01:09:35.828516 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.811512 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 01:09:35.828516 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.811520 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 01:09:35.828516 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:35.812021 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 01:09:35.828516 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:35.812063 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-179.us-west-2.compute.internal\" not found" Nov 04 01:09:35.828516 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.819045 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 01:09:35.828516 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.820093 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 01:09:35.828516 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.820112 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 01:09:35.828516 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.820149 2580 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 01:09:35.828516 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:35.820182 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 04 01:09:35.877161 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:35.877137 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-179.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 04 01:09:35.878247 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:35.877125 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-179.us-west-2.compute.internal.18e4b2486d79fdb4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-179.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-179.us-west-2.compute.internal,},FirstTimestamp:2026-11-04 01:09:35.74440082 +0000 UTC m=+0.420828459,LastTimestamp:2026-11-04 01:09:35.74440082 +0000 UTC m=+0.420828459,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-179.us-west-2.compute.internal,}" Nov 04 01:09:35.911858 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.911796 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 01:09:35.912596 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.912582 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 01:09:35.912665 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.912607 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 01:09:35.912665 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.912627 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 01:09:35.912665 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.912647 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:35.920748 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.920732 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal"] Nov 04 01:09:35.920827 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.920792 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 01:09:35.921560 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.921547 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 01:09:35.921639 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.921571 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 01:09:35.921639 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.921597 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 01:09:35.922134 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.922120 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-179.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 04 01:09:35.923500 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.923487 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 01:09:35.923713 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.923698 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:35.923770 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.923760 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 01:09:35.924159 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.924136 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 01:09:35.924235 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.924165 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 01:09:35.924235 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.924179 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 01:09:35.924538 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.924521 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 01:09:35.924621 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.924547 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 01:09:35.924621 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.924562 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 01:09:35.924621 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:35.924570 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-179.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 04 01:09:35.924621 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:35.924580 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 04 01:09:35.924621 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:35.924598 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 04 01:09:35.928591 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.928575 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:35.928663 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.928612 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 01:09:35.929542 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.929335 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 01:09:35.929542 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.929361 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 01:09:35.929542 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.929383 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 01:09:35.946187 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:35.946170 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-179.us-west-2.compute.internal\" not found" node="ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:35.950366 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:35.950240 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-179.us-west-2.compute.internal\" not found" node="ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:35.954576 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.954560 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/825c4932c3373ac03b157d52d106cbb9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal\" (UID: \"825c4932c3373ac03b157d52d106cbb9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:35.954652 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.954582 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/825c4932c3373ac03b157d52d106cbb9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal\" (UID: \"825c4932c3373ac03b157d52d106cbb9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:35.954652 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.954641 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5be03f35a536944b4cf2058a360cdcb6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal\" (UID: \"5be03f35a536944b4cf2058a360cdcb6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:35.957842 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.957818 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 01:09:35.957907 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:35.957899 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wzx66" Nov 04 01:09:36.046249 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:36.046230 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 01:09:36.054715 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:36.054697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/825c4932c3373ac03b157d52d106cbb9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal\" (UID: \"825c4932c3373ac03b157d52d106cbb9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:36.054801 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:36.054719 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/825c4932c3373ac03b157d52d106cbb9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal\" (UID: \"825c4932c3373ac03b157d52d106cbb9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:36.054801 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:36.054735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5be03f35a536944b4cf2058a360cdcb6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal\" (UID: \"5be03f35a536944b4cf2058a360cdcb6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:36.054801 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:36.054759 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5be03f35a536944b4cf2058a360cdcb6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal\" (UID: \"5be03f35a536944b4cf2058a360cdcb6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:36.054917 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:36.054795 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/825c4932c3373ac03b157d52d106cbb9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal\" (UID: \"825c4932c3373ac03b157d52d106cbb9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:36.054917 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:36.054794 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/825c4932c3373ac03b157d52d106cbb9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal\" (UID: \"825c4932c3373ac03b157d52d106cbb9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:36.055293 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:36.055280 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:36.055324 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:36.055305 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-179.us-west-2.compute.internal\": node \"ip-10-0-138-179.us-west-2.compute.internal\" not found" Nov 04 01:09:36.173492 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:36.173449 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-179.us-west-2.compute.internal\" not found" Nov 04 01:09:36.250019 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:36.250001 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:36.254803 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:36.254778 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:36.273575 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:36.273556 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-179.us-west-2.compute.internal\" not found" Nov 04 01:09:36.374367 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:36.374347 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-179.us-west-2.compute.internal\" not found" Nov 04 01:09:36.474770 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:36.474718 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-179.us-west-2.compute.internal\" not found" Nov 04 01:09:36.575767 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:36.575741 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-179.us-west-2.compute.internal\" not found" Nov 04 01:09:36.676699 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:36.676672 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-179.us-west-2.compute.internal\" not found" Nov 04 01:09:36.677861 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:36.677845 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 04 01:09:36.677964 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:36.677940 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 01:09:36.678048 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:36.677962 2580 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a3e660436c5ac4b748d5d3f99dae8da8-e741eb2aa4dd6e95.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-138-179.us-west-2.compute.internal.18e4b2486f1b1097\": read tcp 10.0.138.179:54454->3.147.153.216:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-138-179.us-west-2.compute.internal.18e4b2486f1b1097 default 4864 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-179.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-179.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-179.us-west-2.compute.internal,},FirstTimestamp:2026-11-04 01:09:35 +0000 UTC,LastTimestamp:2026-11-04 01:09:35.921559451 +0000 UTC m=+0.597987090,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-179.us-west-2.compute.internal,}" Nov 04 01:09:36.777562 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:36.777519 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-179.us-west-2.compute.internal\" not found" Nov 04 01:09:36.878962 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:36.878940 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-179.us-west-2.compute.internal\" not found" Nov 04 01:09:36.945949 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:36.945931 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 01:09:36.946291 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:36.946273 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 01:09:36.952857 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:36.952838 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:36.962865 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:36.962839 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 01:04:36 +0000 UTC" deadline="2028-05-15 08:10:00.557800804 +0000 UTC" Nov 04 01:09:36.962865 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:36.962861 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13399h0m23.594942686s" Nov 04 01:09:37.023361 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.023341 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 01:09:37.025283 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.025269 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:09:37.088947 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.088901 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 01:09:37.144425 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.144393 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 01:09:37.713651 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:37.713613 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be03f35a536944b4cf2058a360cdcb6.slice/crio-12b8e2e7d0b1cfe4c92e5f13d3fe78740de4021fc7f21746a5bee63bba5a4ec4 WatchSource:0}: Error finding container 12b8e2e7d0b1cfe4c92e5f13d3fe78740de4021fc7f21746a5bee63bba5a4ec4: Status 404 returned error can't find the container with id 12b8e2e7d0b1cfe4c92e5f13d3fe78740de4021fc7f21746a5bee63bba5a4ec4 Nov 04 01:09:37.720179 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:37.720155 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825c4932c3373ac03b157d52d106cbb9.slice/crio-13d5686dc21a2275c049fb9f51447a8f8a3652ae89ee45e5e95bc5efead89a01 WatchSource:0}: Error finding container 13d5686dc21a2275c049fb9f51447a8f8a3652ae89ee45e5e95bc5efead89a01: Status 404 returned error can't find the container with id 13d5686dc21a2275c049fb9f51447a8f8a3652ae89ee45e5e95bc5efead89a01 Nov 04 01:09:37.721841 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.721824 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:09:37.730602 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.730588 2580 apiserver.go:52] "Watching apiserver" Nov 04 01:09:37.754215 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.754183 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 04 01:09:37.825343 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.825303 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" event={"ID":"825c4932c3373ac03b157d52d106cbb9","Type":"ContainerStarted","Data":"13d5686dc21a2275c049fb9f51447a8f8a3652ae89ee45e5e95bc5efead89a01"} Nov 04 01:09:37.826302 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.826281 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" event={"ID":"5be03f35a536944b4cf2058a360cdcb6","Type":"ContainerStarted","Data":"12b8e2e7d0b1cfe4c92e5f13d3fe78740de4021fc7f21746a5bee63bba5a4ec4"} Nov 04 01:09:37.895371 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.895326 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 01:09:37.895636 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.895621 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-zfj7b","kube-system/global-pull-secret-syncer-b2jgm","openshift-dns/node-resolver-xsmlx","openshift-multus/multus-additional-cni-plugins-dg8wm","openshift-ovn-kubernetes/ovnkube-node-8dvnf","openshift-cluster-node-tuning-operator/tuned-vdmn8","openshift-network-diagnostics/network-check-target-xqtpx","openshift-network-operator/iptables-alerter-rmvz5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx","openshift-multus/multus-zn87s","kube-system/konnectivity-agent-qvjtd","kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal","openshift-image-registry/node-ca-zzx84"] Nov 04 01:09:37.899684 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.899669 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:37.901895 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.901875 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qvjtd" Nov 04 01:09:37.904141 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.904119 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zzx84" Nov 04 01:09:37.906403 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.906385 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xsmlx" Nov 04 01:09:37.908798 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.908779 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:37.911620 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.911600 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.914067 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.914051 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rmvz5" Nov 04 01:09:37.917170 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.917010 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:37.917170 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:37.917126 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:09:37.920096 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.920076 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:37.920185 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:37.920130 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:09:37.922755 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.922737 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zn87s" Nov 04 01:09:37.925202 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.925187 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:37.927727 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.927709 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:37.927799 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:37.927766 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:09:37.953866 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.953849 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 01:09:37.957330 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.957304 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 01:09:37.957414 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.957389 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 01:09:37.957465 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.957420 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 01:09:37.957569 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.957555 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g9sbp\"" Nov 04 01:09:37.958719 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.958702 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 01:09:37.958914 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.958899 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7mrtt\"" Nov 04 01:09:37.958981 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.958971 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 01:09:37.961120 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.961103 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 01:09:37.961199 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.961126 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 01:09:37.961416 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.961401 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 01:09:37.961460 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.961438 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j9t67\"" Nov 04 01:09:37.963312 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.963294 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hz84q\"" Nov 04 01:09:37.963469 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.963457 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 01:09:37.963524 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.963484 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 01:09:37.968429 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968408 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm8jv\" (UniqueName: \"kubernetes.io/projected/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-kube-api-access-pm8jv\") pod \"network-metrics-daemon-zfj7b\" (UID: \"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8\") " pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:37.968542 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968434 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-run-multus-certs\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:37.968542 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968510 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlf5p\" (UniqueName: \"kubernetes.io/projected/03f3058b-0663-48a7-ab67-a41bfe5a2c57-kube-api-access-rlf5p\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:37.968542 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968535 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:37.968660 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968551 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-var-lib-openvswitch\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.968660 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968570 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-run-ovn-kubernetes\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.968660 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968601 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3d08d00-e371-4793-955e-c5b51eef7879-system-cni-dir\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:37.968660 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968632 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3d08d00-e371-4793-955e-c5b51eef7879-cnibin\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:37.968660 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968651 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:37.968803 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-systemd-units\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.968803 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968702 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-cni-binary-copy\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:37.968803 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968736 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-run-k8s-cni-cncf-io\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:37.968803 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-daemon-config\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:37.968803 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968789 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-modprobe-d\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:37.968949 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-kubernetes\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:37.968949 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968843 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:37.968949 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968860 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-sysconfig\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:37.968949 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968890 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f50bc9b-91da-4d92-8553-c9f9476665e4-serviceca\") pod \"node-ca-zzx84\" (UID: \"0f50bc9b-91da-4d92-8553-c9f9476665e4\") " pod="openshift-image-registry/node-ca-zzx84" Nov 04 01:09:37.968949 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968916 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-node-log\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.968949 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968933 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.969145 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968969 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltmvj\" (UniqueName: \"kubernetes.io/projected/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-kube-api-access-ltmvj\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:37.969145 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.968991 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-kubelet\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.969145 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969009 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3d08d00-e371-4793-955e-c5b51eef7879-cni-binary-copy\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:37.969145 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969044 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3d08d00-e371-4793-955e-c5b51eef7879-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:37.969145 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969063 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5b0a3f1-5189-4572-9765-20eb3960fbae-host-slash\") pod \"iptables-alerter-rmvz5\" (UID: \"c5b0a3f1-5189-4572-9765-20eb3960fbae\") " pod="openshift-network-operator/iptables-alerter-rmvz5" Nov 04 01:09:37.969145 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969092 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-systemd\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:37.969145 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-var-lib-kubelet\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:37.969343 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969155 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-hostroot\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:37.969343 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969172 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-host\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:37.969343 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-ovnkube-config\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.969343 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969211 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3d08d00-e371-4793-955e-c5b51eef7879-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:37.969343 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969230 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-var-lib-cni-multus\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:37.969343 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-slash\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.969343 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-run-openvswitch\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.969343 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969278 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-log-socket\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.969343 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969293 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-ovnkube-script-lib\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.969343 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5b0a3f1-5189-4572-9765-20eb3960fbae-iptables-alerter-script\") pod \"iptables-alerter-rmvz5\" (UID: \"c5b0a3f1-5189-4572-9765-20eb3960fbae\") " pod="openshift-network-operator/iptables-alerter-rmvz5" Nov 04 01:09:37.969343 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969332 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-system-cni-dir\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:37.969691 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-run-netns\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:37.969691 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-conf-dir\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:37.969691 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969404 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-etc-kubernetes\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:37.969691 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969439 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a06edce6-baf3-40bc-9d0e-a6e71c393165-konnectivity-ca\") pod \"konnectivity-agent-qvjtd\" (UID: \"a06edce6-baf3-40bc-9d0e-a6e71c393165\") " pod="kube-system/konnectivity-agent-qvjtd" Nov 04 01:09:37.969691 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969458 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-run-netns\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.969691 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969472 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-env-overrides\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.969691 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969487 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxlfx\" (UniqueName: \"kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx\") pod \"network-check-target-xqtpx\" (UID: \"42f02121-70f2-43d1-be9d-2a81425ec97d\") " pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:37.969691 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969501 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-socket-dir-parent\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:37.969691 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969515 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-var-lib-cni-bin\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:37.969691 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969535 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slglc\" (UniqueName: \"kubernetes.io/projected/ffb55ed6-b225-4db5-a7fb-66a10868befc-kube-api-access-slglc\") pod \"node-resolver-xsmlx\" (UID: \"ffb55ed6-b225-4db5-a7fb-66a10868befc\") " pod="openshift-dns/node-resolver-xsmlx" Nov 04 01:09:37.969691 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969559 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs\") pod \"network-metrics-daemon-zfj7b\" (UID: \"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8\") " pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:37.969691 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969579 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-sysctl-conf\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:37.969691 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969601 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-tuned\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:37.969691 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969630 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03f3058b-0663-48a7-ab67-a41bfe5a2c57-tmp\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:37.969691 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969645 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-run-ovn\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.969691 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969686 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-cni-netd\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.970170 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969713 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-sysctl-d\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:37.970170 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969732 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ae06262-6e59-4775-bedc-5305ace1fae6-dbus\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:37.970170 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969771 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-245fq\" (UniqueName: \"kubernetes.io/projected/c5b0a3f1-5189-4572-9765-20eb3960fbae-kube-api-access-245fq\") pod \"iptables-alerter-rmvz5\" (UID: \"c5b0a3f1-5189-4572-9765-20eb3960fbae\") " pod="openshift-network-operator/iptables-alerter-rmvz5" Nov 04 01:09:37.970170 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969799 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-sys\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:37.970170 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969817 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-registration-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:37.970170 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969840 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a06edce6-baf3-40bc-9d0e-a6e71c393165-agent-certs\") pod \"konnectivity-agent-qvjtd\" (UID: \"a06edce6-baf3-40bc-9d0e-a6e71c393165\") " pod="kube-system/konnectivity-agent-qvjtd" Nov 04 01:09:37.970170 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969865 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb55ed6-b225-4db5-a7fb-66a10868befc-tmp-dir\") pod \"node-resolver-xsmlx\" (UID: \"ffb55ed6-b225-4db5-a7fb-66a10868befc\") " pod="openshift-dns/node-resolver-xsmlx" Nov 04 01:09:37.970170 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-cni-dir\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:37.970170 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969915 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpmgk\" (UniqueName: \"kubernetes.io/projected/82966365-6bec-467d-9279-12ac45dd01e2-kube-api-access-fpmgk\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:37.970170 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969941 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-cni-bin\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.970170 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.969991 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3d08d00-e371-4793-955e-c5b51eef7879-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:37.970170 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.970013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-run\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:37.970170 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.970030 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-cnibin\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:37.970170 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.970048 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-var-lib-kubelet\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:37.970170 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.970064 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-device-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:37.970170 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.970086 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk7ck\" (UniqueName: \"kubernetes.io/projected/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-kube-api-access-bk7ck\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.970747 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.970113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-lib-modules\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:37.970747 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.970146 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ae06262-6e59-4775-bedc-5305ace1fae6-kubelet-config\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:37.970747 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.970167 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-socket-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:37.970747 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.970182 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-sys-fs\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:37.970747 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.970204 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhhtl\" (UniqueName: \"kubernetes.io/projected/0f50bc9b-91da-4d92-8553-c9f9476665e4-kube-api-access-zhhtl\") pod \"node-ca-zzx84\" (UID: \"0f50bc9b-91da-4d92-8553-c9f9476665e4\") " pod="openshift-image-registry/node-ca-zzx84" Nov 04 01:09:37.970747 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.970247 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ffb55ed6-b225-4db5-a7fb-66a10868befc-hosts-file\") pod \"node-resolver-xsmlx\" (UID: \"ffb55ed6-b225-4db5-a7fb-66a10868befc\") " pod="openshift-dns/node-resolver-xsmlx" Nov 04 01:09:37.970747 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.970265 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-run-systemd\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.970747 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.970280 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-etc-openvswitch\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.970747 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.970295 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-ovn-node-metrics-cert\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:37.970747 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.970312 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f50bc9b-91da-4d92-8553-c9f9476665e4-host\") pod \"node-ca-zzx84\" (UID: \"0f50bc9b-91da-4d92-8553-c9f9476665e4\") " pod="openshift-image-registry/node-ca-zzx84" Nov 04 01:09:37.970747 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:37.970327 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8xv5\" (UniqueName: \"kubernetes.io/projected/c3d08d00-e371-4793-955e-c5b51eef7879-kube-api-access-v8xv5\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:38.006699 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.006681 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 01:09:38.007040 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.007022 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 01:09:38.007123 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.007065 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 01:09:38.007256 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.007226 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 01:09:38.007356 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.007262 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 01:09:38.007559 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.007544 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 01:09:38.007810 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.007792 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 01:09:38.007907 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.007837 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 01:09:38.007907 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.007849 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4z2qg\"" Nov 04 01:09:38.007907 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.007868 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 01:09:38.007907 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.007840 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:09:38.007907 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.007850 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 01:09:38.008109 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.008092 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 01:09:38.008166 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.008153 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 01:09:38.008209 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.008186 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rjcnb\"" Nov 04 01:09:38.008399 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.008386 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q5jlj\"" Nov 04 01:09:38.012030 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.012015 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 01:09:38.012212 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.012199 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:09:38.012416 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.012390 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 01:09:38.012416 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.012395 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-prqm6\"" Nov 04 01:09:38.012568 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.012523 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d8jdw\"" Nov 04 01:09:38.012568 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.012532 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 01:09:38.012635 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.012591 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 01:09:38.012635 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.012621 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q5kf6" Nov 04 01:09:38.071160 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071130 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ae06262-6e59-4775-bedc-5305ace1fae6-kubelet-config\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:38.071254 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071192 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ae06262-6e59-4775-bedc-5305ace1fae6-kubelet-config\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:38.071254 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071219 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-socket-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:38.071364 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071260 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-sys-fs\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:38.071364 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071289 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhhtl\" (UniqueName: \"kubernetes.io/projected/0f50bc9b-91da-4d92-8553-c9f9476665e4-kube-api-access-zhhtl\") pod \"node-ca-zzx84\" (UID: \"0f50bc9b-91da-4d92-8553-c9f9476665e4\") " pod="openshift-image-registry/node-ca-zzx84" Nov 04 01:09:38.071364 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ffb55ed6-b225-4db5-a7fb-66a10868befc-hosts-file\") pod \"node-resolver-xsmlx\" (UID: \"ffb55ed6-b225-4db5-a7fb-66a10868befc\") " pod="openshift-dns/node-resolver-xsmlx" Nov 04 01:09:38.071364 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071326 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-sys-fs\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:38.071364 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071334 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-socket-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:38.071364 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-run-systemd\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.071642 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071388 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-run-systemd\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.071642 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-etc-openvswitch\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.071642 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071455 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-ovn-node-metrics-cert\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.071642 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f50bc9b-91da-4d92-8553-c9f9476665e4-host\") pod \"node-ca-zzx84\" (UID: \"0f50bc9b-91da-4d92-8553-c9f9476665e4\") " pod="openshift-image-registry/node-ca-zzx84" Nov 04 01:09:38.071642 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071453 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-etc-openvswitch\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.071642 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071487 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8xv5\" (UniqueName: \"kubernetes.io/projected/c3d08d00-e371-4793-955e-c5b51eef7879-kube-api-access-v8xv5\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:38.071642 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071511 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ffb55ed6-b225-4db5-a7fb-66a10868befc-hosts-file\") pod \"node-resolver-xsmlx\" (UID: \"ffb55ed6-b225-4db5-a7fb-66a10868befc\") " pod="openshift-dns/node-resolver-xsmlx" Nov 04 01:09:38.071642 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071550 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f50bc9b-91da-4d92-8553-c9f9476665e4-host\") pod \"node-ca-zzx84\" (UID: \"0f50bc9b-91da-4d92-8553-c9f9476665e4\") " pod="openshift-image-registry/node-ca-zzx84" Nov 04 01:09:38.071642 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm8jv\" (UniqueName: \"kubernetes.io/projected/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-kube-api-access-pm8jv\") pod \"network-metrics-daemon-zfj7b\" (UID: \"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8\") " pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:38.071642 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071595 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-run-multus-certs\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.071642 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlf5p\" (UniqueName: \"kubernetes.io/projected/03f3058b-0663-48a7-ab67-a41bfe5a2c57-kube-api-access-rlf5p\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.072180 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071671 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:38.072180 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-var-lib-openvswitch\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.072180 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071732 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-run-multus-certs\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.072180 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071747 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-run-ovn-kubernetes\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.072180 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3d08d00-e371-4793-955e-c5b51eef7879-system-cni-dir\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:38.072180 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071768 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:38.072180 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071803 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3d08d00-e371-4793-955e-c5b51eef7879-cnibin\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:38.072180 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071818 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 01:09:38.072180 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071827 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-var-lib-openvswitch\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.072180 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071831 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:38.072180 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071853 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-run-ovn-kubernetes\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.072180 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071870 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3d08d00-e371-4793-955e-c5b51eef7879-cnibin\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:38.072180 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-systemd-units\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.072180 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071910 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-cni-binary-copy\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.072180 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071928 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-run-k8s-cni-cncf-io\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.072180 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071916 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3d08d00-e371-4793-955e-c5b51eef7879-system-cni-dir\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:38.072180 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071948 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:38.073017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.071960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-daemon-config\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.073017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072068 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-modprobe-d\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.073017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072074 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-systemd-units\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.073017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-kubernetes\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.073017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072097 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-run-k8s-cni-cncf-io\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.073017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:38.073017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-sysconfig\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.073017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072179 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f50bc9b-91da-4d92-8553-c9f9476665e4-serviceca\") pod \"node-ca-zzx84\" (UID: \"0f50bc9b-91da-4d92-8553-c9f9476665e4\") " pod="openshift-image-registry/node-ca-zzx84" Nov 04 01:09:38.073017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072199 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-node-log\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.073017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072237 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-kubernetes\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.073017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.073017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072279 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltmvj\" (UniqueName: \"kubernetes.io/projected/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-kube-api-access-ltmvj\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.073017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072274 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-modprobe-d\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.073017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-kubelet\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.073017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3d08d00-e371-4793-955e-c5b51eef7879-cni-binary-copy\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:38.073017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072389 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3d08d00-e371-4793-955e-c5b51eef7879-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:38.073017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072414 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5b0a3f1-5189-4572-9765-20eb3960fbae-host-slash\") pod \"iptables-alerter-rmvz5\" (UID: \"c5b0a3f1-5189-4572-9765-20eb3960fbae\") " pod="openshift-network-operator/iptables-alerter-rmvz5" Nov 04 01:09:38.073766 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072459 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-systemd\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.073766 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072492 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-var-lib-kubelet\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.073766 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072516 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-hostroot\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.073766 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072554 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-host\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.073766 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-ovnkube-config\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.073766 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072612 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3d08d00-e371-4793-955e-c5b51eef7879-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:38.073766 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-var-lib-cni-multus\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.073766 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072660 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-slash\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.073766 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072669 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f50bc9b-91da-4d92-8553-c9f9476665e4-serviceca\") pod \"node-ca-zzx84\" (UID: \"0f50bc9b-91da-4d92-8553-c9f9476665e4\") " pod="openshift-image-registry/node-ca-zzx84" Nov 04 01:09:38.073766 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:38.072728 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 01:09:38.073766 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072749 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-slash\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.073766 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072734 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-daemon-config\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.073766 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072768 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-cni-binary-copy\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.073766 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072787 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-run-openvswitch\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.073766 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:38.072813 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret podName:3ae06262-6e59-4775-bedc-5305ace1fae6 nodeName:}" failed. No retries permitted until 2026-11-04 01:09:38.572780381 +0000 UTC m=+3.249208022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret") pod "global-pull-secret-syncer-b2jgm" (UID: "3ae06262-6e59-4775-bedc-5305ace1fae6") : object "kube-system"/"original-pull-secret" not registered Nov 04 01:09:38.073766 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072813 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-run-openvswitch\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.073766 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072809 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.074670 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072836 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-var-lib-kubelet\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.074670 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072857 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-log-socket\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.074670 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072882 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-hostroot\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.074670 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-ovnkube-script-lib\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.074670 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072914 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5b0a3f1-5189-4572-9765-20eb3960fbae-iptables-alerter-script\") pod \"iptables-alerter-rmvz5\" (UID: \"c5b0a3f1-5189-4572-9765-20eb3960fbae\") " pod="openshift-network-operator/iptables-alerter-rmvz5" Nov 04 01:09:38.074670 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072916 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-log-socket\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.074670 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072942 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-host\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.074670 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.072983 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-kubelet\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.074670 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.073284 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-sysconfig\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.074670 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.073421 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-system-cni-dir\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.074670 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.073500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-run-netns\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.074670 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.073529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-conf-dir\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.074670 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.073565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-etc-kubernetes\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.074670 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.073599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a06edce6-baf3-40bc-9d0e-a6e71c393165-konnectivity-ca\") pod \"konnectivity-agent-qvjtd\" (UID: \"a06edce6-baf3-40bc-9d0e-a6e71c393165\") " pod="kube-system/konnectivity-agent-qvjtd" Nov 04 01:09:38.074670 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.073608 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-var-lib-cni-multus\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.074670 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.073634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-run-netns\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.074670 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.073653 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-node-log\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.074670 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.073667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-env-overrides\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.075508 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.073738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxlfx\" (UniqueName: \"kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx\") pod \"network-check-target-xqtpx\" (UID: \"42f02121-70f2-43d1-be9d-2a81425ec97d\") " pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:38.075508 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.073798 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-socket-dir-parent\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.075508 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.073797 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5b0a3f1-5189-4572-9765-20eb3960fbae-iptables-alerter-script\") pod \"iptables-alerter-rmvz5\" (UID: \"c5b0a3f1-5189-4572-9765-20eb3960fbae\") " pod="openshift-network-operator/iptables-alerter-rmvz5" Nov 04 01:09:38.075508 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.073843 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-conf-dir\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.075508 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.073920 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-system-cni-dir\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.075508 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.073917 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3d08d00-e371-4793-955e-c5b51eef7879-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:38.075508 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074004 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-ovnkube-config\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.075508 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074022 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5b0a3f1-5189-4572-9765-20eb3960fbae-host-slash\") pod \"iptables-alerter-rmvz5\" (UID: \"c5b0a3f1-5189-4572-9765-20eb3960fbae\") " pod="openshift-network-operator/iptables-alerter-rmvz5" Nov 04 01:09:38.075508 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-var-lib-cni-bin\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.075508 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slglc\" (UniqueName: \"kubernetes.io/projected/ffb55ed6-b225-4db5-a7fb-66a10868befc-kube-api-access-slglc\") pod \"node-resolver-xsmlx\" (UID: \"ffb55ed6-b225-4db5-a7fb-66a10868befc\") " pod="openshift-dns/node-resolver-xsmlx" Nov 04 01:09:38.075508 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074194 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-env-overrides\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.075508 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074202 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs\") pod \"network-metrics-daemon-zfj7b\" (UID: \"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8\") " pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:38.075508 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074248 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-sysctl-conf\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.075508 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074288 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-tuned\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.075508 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074294 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-run-netns\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.075508 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074342 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03f3058b-0663-48a7-ab67-a41bfe5a2c57-tmp\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.075508 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074404 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-socket-dir-parent\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.076329 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074414 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-run-ovn\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.076329 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074448 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-run-netns\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.076329 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074484 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-cni-netd\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.076329 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074511 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-run-ovn\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.076329 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074560 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-etc-kubernetes\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.076329 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074576 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-cni-netd\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.076329 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074584 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3d08d00-e371-4793-955e-c5b51eef7879-cni-binary-copy\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:38.076329 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074619 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-sysctl-d\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.076329 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074654 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-var-lib-cni-bin\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.076329 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ae06262-6e59-4775-bedc-5305ace1fae6-dbus\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:38.076329 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074685 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-systemd\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.076329 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074718 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-245fq\" (UniqueName: \"kubernetes.io/projected/c5b0a3f1-5189-4572-9765-20eb3960fbae-kube-api-access-245fq\") pod \"iptables-alerter-rmvz5\" (UID: \"c5b0a3f1-5189-4572-9765-20eb3960fbae\") " pod="openshift-network-operator/iptables-alerter-rmvz5" Nov 04 01:09:38.076329 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074747 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3d08d00-e371-4793-955e-c5b51eef7879-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:38.076329 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074773 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-sysctl-d\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.076329 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074803 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-sysctl-conf\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.076329 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-sys\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.076329 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074904 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a06edce6-baf3-40bc-9d0e-a6e71c393165-konnectivity-ca\") pod \"konnectivity-agent-qvjtd\" (UID: \"a06edce6-baf3-40bc-9d0e-a6e71c393165\") " pod="kube-system/konnectivity-agent-qvjtd" Nov 04 01:09:38.076329 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.074965 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-registration-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:38.077228 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.075001 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a06edce6-baf3-40bc-9d0e-a6e71c393165-agent-certs\") pod \"konnectivity-agent-qvjtd\" (UID: \"a06edce6-baf3-40bc-9d0e-a6e71c393165\") " pod="kube-system/konnectivity-agent-qvjtd" Nov 04 01:09:38.077228 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.075108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb55ed6-b225-4db5-a7fb-66a10868befc-tmp-dir\") pod \"node-resolver-xsmlx\" (UID: \"ffb55ed6-b225-4db5-a7fb-66a10868befc\") " pod="openshift-dns/node-resolver-xsmlx" Nov 04 01:09:38.077228 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.075146 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-sys\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.077228 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.075164 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-cni-dir\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.077228 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.075256 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-registration-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:38.077228 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.075311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpmgk\" (UniqueName: \"kubernetes.io/projected/82966365-6bec-467d-9279-12ac45dd01e2-kube-api-access-fpmgk\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:38.077228 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.075362 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-cni-bin\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.077228 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:38.075366 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:09:38.077228 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.075395 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3d08d00-e371-4793-955e-c5b51eef7879-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:38.077228 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:38.075436 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs podName:fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8 nodeName:}" failed. No retries permitted until 2026-11-04 01:09:38.575410708 +0000 UTC m=+3.251838354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs") pod "network-metrics-daemon-zfj7b" (UID: "fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:09:38.077228 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.076236 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-run\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.077228 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.076252 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-cni-bin\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.077228 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.076303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-cnibin\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.077228 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.076334 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-var-lib-kubelet\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.077228 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.076384 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-device-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:38.077228 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.076405 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3d08d00-e371-4793-955e-c5b51eef7879-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:38.077228 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.076439 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk7ck\" (UniqueName: \"kubernetes.io/projected/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-kube-api-access-bk7ck\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.078076 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.076474 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-lib-modules\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.078076 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.076632 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-lib-modules\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.078076 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.075549 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ae06262-6e59-4775-bedc-5305ace1fae6-dbus\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:38.078076 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.076712 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-run\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.078076 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.075659 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-cni-dir\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.078076 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.076788 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-cnibin\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.078076 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.076847 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-var-lib-kubelet\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.078076 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.076913 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-device-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:38.078076 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.077240 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb55ed6-b225-4db5-a7fb-66a10868befc-tmp-dir\") pod \"node-resolver-xsmlx\" (UID: \"ffb55ed6-b225-4db5-a7fb-66a10868befc\") " pod="openshift-dns/node-resolver-xsmlx" Nov 04 01:09:38.078076 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.077446 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-tuned\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.078076 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.077541 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-ovnkube-script-lib\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.078076 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.077666 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03f3058b-0663-48a7-ab67-a41bfe5a2c57-tmp\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.078076 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.077676 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-ovn-node-metrics-cert\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.078933 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.078899 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a06edce6-baf3-40bc-9d0e-a6e71c393165-agent-certs\") pod \"konnectivity-agent-qvjtd\" (UID: \"a06edce6-baf3-40bc-9d0e-a6e71c393165\") " pod="kube-system/konnectivity-agent-qvjtd" Nov 04 01:09:38.099358 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.099343 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 01:09:38.137713 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:38.137687 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 01:09:38.137713 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:38.137713 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 01:09:38.137856 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:38.137728 2580 projected.go:196] Error preparing data for projected volume kube-api-access-sxlfx for pod openshift-network-diagnostics/network-check-target-xqtpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:09:38.137856 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:38.137802 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx podName:42f02121-70f2-43d1-be9d-2a81425ec97d nodeName:}" failed. No retries permitted until 2026-11-04 01:09:38.637783611 +0000 UTC m=+3.314211256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sxlfx" (UniqueName: "kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx") pod "network-check-target-xqtpx" (UID: "42f02121-70f2-43d1-be9d-2a81425ec97d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:09:38.138282 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.138259 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhhtl\" (UniqueName: \"kubernetes.io/projected/0f50bc9b-91da-4d92-8553-c9f9476665e4-kube-api-access-zhhtl\") pod \"node-ca-zzx84\" (UID: \"0f50bc9b-91da-4d92-8553-c9f9476665e4\") " pod="openshift-image-registry/node-ca-zzx84" Nov 04 01:09:38.138388 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.138367 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlf5p\" (UniqueName: \"kubernetes.io/projected/03f3058b-0663-48a7-ab67-a41bfe5a2c57-kube-api-access-rlf5p\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.138571 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.138554 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm8jv\" (UniqueName: \"kubernetes.io/projected/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-kube-api-access-pm8jv\") pod \"network-metrics-daemon-zfj7b\" (UID: \"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8\") " pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:38.138725 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.138707 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltmvj\" (UniqueName: \"kubernetes.io/projected/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-kube-api-access-ltmvj\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.139259 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.139238 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8xv5\" (UniqueName: \"kubernetes.io/projected/c3d08d00-e371-4793-955e-c5b51eef7879-kube-api-access-v8xv5\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:38.140814 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.140786 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slglc\" (UniqueName: \"kubernetes.io/projected/ffb55ed6-b225-4db5-a7fb-66a10868befc-kube-api-access-slglc\") pod \"node-resolver-xsmlx\" (UID: \"ffb55ed6-b225-4db5-a7fb-66a10868befc\") " pod="openshift-dns/node-resolver-xsmlx" Nov 04 01:09:38.140906 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.140884 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-245fq\" (UniqueName: \"kubernetes.io/projected/c5b0a3f1-5189-4572-9765-20eb3960fbae-kube-api-access-245fq\") pod \"iptables-alerter-rmvz5\" (UID: \"c5b0a3f1-5189-4572-9765-20eb3960fbae\") " pod="openshift-network-operator/iptables-alerter-rmvz5" Nov 04 01:09:38.141681 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.141662 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpmgk\" (UniqueName: \"kubernetes.io/projected/82966365-6bec-467d-9279-12ac45dd01e2-kube-api-access-fpmgk\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:38.142355 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.142339 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk7ck\" (UniqueName: \"kubernetes.io/projected/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-kube-api-access-bk7ck\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.215709 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.215663 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:09:38.217779 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.217756 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qvjtd" Nov 04 01:09:38.226718 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:38.226689 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82966365_6bec_467d_9279_12ac45dd01e2.slice/crio-57632676aacede02ef6f25328a842b6cdddff2f9f7290826b4767f5643f043f0 WatchSource:0}: Error finding container 57632676aacede02ef6f25328a842b6cdddff2f9f7290826b4767f5643f043f0: Status 404 returned error can't find the container with id 57632676aacede02ef6f25328a842b6cdddff2f9f7290826b4767f5643f043f0 Nov 04 01:09:38.227536 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.227522 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zzx84" Nov 04 01:09:38.229353 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:38.229332 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06edce6_baf3_40bc_9d0e_a6e71c393165.slice/crio-98da5efe82e0b952f83c1d1541ad31ac5ab87b519f7a25be001d0ab695443267 WatchSource:0}: Error finding container 98da5efe82e0b952f83c1d1541ad31ac5ab87b519f7a25be001d0ab695443267: Status 404 returned error can't find the container with id 98da5efe82e0b952f83c1d1541ad31ac5ab87b519f7a25be001d0ab695443267 Nov 04 01:09:38.232743 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.232722 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xsmlx" Nov 04 01:09:38.237558 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:38.237535 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f50bc9b_91da_4d92_8553_c9f9476665e4.slice/crio-98172af2c8dfd62512835ca75a3182e74da317adbe908f18a05b0e0c39af5bfa WatchSource:0}: Error finding container 98172af2c8dfd62512835ca75a3182e74da317adbe908f18a05b0e0c39af5bfa: Status 404 returned error can't find the container with id 98172af2c8dfd62512835ca75a3182e74da317adbe908f18a05b0e0c39af5bfa Nov 04 01:09:38.242567 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:38.242548 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb55ed6_b225_4db5_a7fb_66a10868befc.slice/crio-738361f938aa7c6d4c807186a4dced11bcc6dbea29c8580aeff49fc3f28ff4f2 WatchSource:0}: Error finding container 738361f938aa7c6d4c807186a4dced11bcc6dbea29c8580aeff49fc3f28ff4f2: Status 404 returned error can't find the container with id 738361f938aa7c6d4c807186a4dced11bcc6dbea29c8580aeff49fc3f28ff4f2 Nov 04 01:09:38.255837 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.255815 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rmvz5" Nov 04 01:09:38.260702 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.260688 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:09:38.263039 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.263022 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zn87s" Nov 04 01:09:38.264427 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:38.264399 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b0a3f1_5189_4572_9765_20eb3960fbae.slice/crio-18cf5ce0a486bf5b432b3279552d68791ee83754ac6cc2a8bf87576938774556 WatchSource:0}: Error finding container 18cf5ce0a486bf5b432b3279552d68791ee83754ac6cc2a8bf87576938774556: Status 404 returned error can't find the container with id 18cf5ce0a486bf5b432b3279552d68791ee83754ac6cc2a8bf87576938774556 Nov 04 01:09:38.269228 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.269123 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:09:38.269407 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:38.269375 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d08d00_e371_4793_955e_c5b51eef7879.slice/crio-97eefcf4cb3d26c1db3a3ea4102ab8d0e9226c78b7451d734136ca782bf547f8 WatchSource:0}: Error finding container 97eefcf4cb3d26c1db3a3ea4102ab8d0e9226c78b7451d734136ca782bf547f8: Status 404 returned error can't find the container with id 97eefcf4cb3d26c1db3a3ea4102ab8d0e9226c78b7451d734136ca782bf547f8 Nov 04 01:09:38.271276 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:38.271260 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce3f5ec_19d0_41a4_8e27_454b22fc9ed5.slice/crio-4f47617b789c3b3cb03076d6552d01b677d953d5f1021e70bb844f965e7f17cf WatchSource:0}: Error finding container 4f47617b789c3b3cb03076d6552d01b677d953d5f1021e70bb844f965e7f17cf: Status 404 returned error can't find the container with id 4f47617b789c3b3cb03076d6552d01b677d953d5f1021e70bb844f965e7f17cf Nov 04 01:09:38.275817 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.275796 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:09:38.278649 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:38.278630 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f3058b_0663_48a7_ab67_a41bfe5a2c57.slice/crio-79659f857916f206da3983afc668273bca66cf7cca19767c2c79eaeeafca5d93 WatchSource:0}: Error finding container 79659f857916f206da3983afc668273bca66cf7cca19767c2c79eaeeafca5d93: Status 404 returned error can't find the container with id 79659f857916f206da3983afc668273bca66cf7cca19767c2c79eaeeafca5d93 Nov 04 01:09:38.284582 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:09:38.284560 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bd09a4_6d44_49ae_9bfa_837e2e5bf915.slice/crio-acf4fc0ba80c720332e88d492e223ff678c13b3e7604db81bd64148c02b40836 WatchSource:0}: Error finding container acf4fc0ba80c720332e88d492e223ff678c13b3e7604db81bd64148c02b40836: Status 404 returned error can't find the container with id acf4fc0ba80c720332e88d492e223ff678c13b3e7604db81bd64148c02b40836 Nov 04 01:09:38.580343 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.580283 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:38.580343 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.580325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs\") pod \"network-metrics-daemon-zfj7b\" (UID: \"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8\") " pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:38.580496 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:38.580406 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:09:38.580496 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:38.580431 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 01:09:38.580496 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:38.580464 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs podName:fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8 nodeName:}" failed. No retries permitted until 2026-11-04 01:09:39.580448752 +0000 UTC m=+4.256876393 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs") pod "network-metrics-daemon-zfj7b" (UID: "fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:09:38.580496 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:38.580488 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret podName:3ae06262-6e59-4775-bedc-5305ace1fae6 nodeName:}" failed. No retries permitted until 2026-11-04 01:09:39.58047605 +0000 UTC m=+4.256903684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret") pod "global-pull-secret-syncer-b2jgm" (UID: "3ae06262-6e59-4775-bedc-5305ace1fae6") : object "kube-system"/"original-pull-secret" not registered Nov 04 01:09:38.681363 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.681333 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxlfx\" (UniqueName: \"kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx\") pod \"network-check-target-xqtpx\" (UID: \"42f02121-70f2-43d1-be9d-2a81425ec97d\") " pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:38.681503 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:38.681430 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 01:09:38.681503 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:38.681460 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 01:09:38.681503 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:38.681468 2580 projected.go:196] Error preparing data for projected volume kube-api-access-sxlfx for pod openshift-network-diagnostics/network-check-target-xqtpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:09:38.681612 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:38.681508 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx podName:42f02121-70f2-43d1-be9d-2a81425ec97d nodeName:}" failed. No retries permitted until 2026-11-04 01:09:39.681497978 +0000 UTC m=+4.357925606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sxlfx" (UniqueName: "kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx") pod "network-check-target-xqtpx" (UID: "42f02121-70f2-43d1-be9d-2a81425ec97d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:09:38.828478 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.828452 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zzx84" event={"ID":"0f50bc9b-91da-4d92-8553-c9f9476665e4","Type":"ContainerStarted","Data":"98172af2c8dfd62512835ca75a3182e74da317adbe908f18a05b0e0c39af5bfa"} Nov 04 01:09:38.829820 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.829801 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" event={"ID":"89bd09a4-6d44-49ae-9bfa-837e2e5bf915","Type":"ContainerStarted","Data":"acf4fc0ba80c720332e88d492e223ff678c13b3e7604db81bd64148c02b40836"} Nov 04 01:09:38.831062 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.831010 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zn87s" event={"ID":"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5","Type":"ContainerStarted","Data":"4f47617b789c3b3cb03076d6552d01b677d953d5f1021e70bb844f965e7f17cf"} Nov 04 01:09:38.832213 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.832191 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qvjtd" event={"ID":"a06edce6-baf3-40bc-9d0e-a6e71c393165","Type":"ContainerStarted","Data":"98da5efe82e0b952f83c1d1541ad31ac5ab87b519f7a25be001d0ab695443267"} Nov 04 01:09:38.833345 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.833306 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" event={"ID":"82966365-6bec-467d-9279-12ac45dd01e2","Type":"ContainerStarted","Data":"57632676aacede02ef6f25328a842b6cdddff2f9f7290826b4767f5643f043f0"} Nov 04 01:09:38.834533 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.834514 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" event={"ID":"03f3058b-0663-48a7-ab67-a41bfe5a2c57","Type":"ContainerStarted","Data":"79659f857916f206da3983afc668273bca66cf7cca19767c2c79eaeeafca5d93"} Nov 04 01:09:38.835582 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.835557 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dg8wm" event={"ID":"c3d08d00-e371-4793-955e-c5b51eef7879","Type":"ContainerStarted","Data":"97eefcf4cb3d26c1db3a3ea4102ab8d0e9226c78b7451d734136ca782bf547f8"} Nov 04 01:09:38.836971 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.836950 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rmvz5" event={"ID":"c5b0a3f1-5189-4572-9765-20eb3960fbae","Type":"ContainerStarted","Data":"18cf5ce0a486bf5b432b3279552d68791ee83754ac6cc2a8bf87576938774556"} Nov 04 01:09:38.838076 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:38.838053 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xsmlx" event={"ID":"ffb55ed6-b225-4db5-a7fb-66a10868befc","Type":"ContainerStarted","Data":"738361f938aa7c6d4c807186a4dced11bcc6dbea29c8580aeff49fc3f28ff4f2"} Nov 04 01:09:39.016165 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:39.016103 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 01:04:38 +0000 UTC" deadline="2028-08-14 09:10:49.00951608 +0000 UTC" Nov 04 01:09:39.016165 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:39.016165 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15584h1m9.993355972s" Nov 04 01:09:39.588001 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:39.587689 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:39.588001 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:39.587746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs\") pod \"network-metrics-daemon-zfj7b\" (UID: \"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8\") " pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:39.588001 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:39.587862 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:09:39.588001 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:39.587908 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs podName:fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8 nodeName:}" failed. No retries permitted until 2026-11-04 01:09:41.587894569 +0000 UTC m=+6.264322210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs") pod "network-metrics-daemon-zfj7b" (UID: "fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:09:39.588001 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:39.587952 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 01:09:39.588001 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:39.587974 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret podName:3ae06262-6e59-4775-bedc-5305ace1fae6 nodeName:}" failed. No retries permitted until 2026-11-04 01:09:41.58796706 +0000 UTC m=+6.264394701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret") pod "global-pull-secret-syncer-b2jgm" (UID: "3ae06262-6e59-4775-bedc-5305ace1fae6") : object "kube-system"/"original-pull-secret" not registered Nov 04 01:09:39.689369 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:39.689024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxlfx\" (UniqueName: \"kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx\") pod \"network-check-target-xqtpx\" (UID: \"42f02121-70f2-43d1-be9d-2a81425ec97d\") " pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:39.689369 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:39.689238 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 01:09:39.689369 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:39.689263 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 01:09:39.689369 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:39.689276 2580 projected.go:196] Error preparing data for projected volume kube-api-access-sxlfx for pod openshift-network-diagnostics/network-check-target-xqtpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:09:39.689369 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:39.689332 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx podName:42f02121-70f2-43d1-be9d-2a81425ec97d nodeName:}" failed. No retries permitted until 2026-11-04 01:09:41.689313782 +0000 UTC m=+6.365741424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sxlfx" (UniqueName: "kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx") pod "network-check-target-xqtpx" (UID: "42f02121-70f2-43d1-be9d-2a81425ec97d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:09:39.823774 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:39.823738 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:39.823960 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:39.823876 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:09:39.824043 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:39.823961 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:39.824095 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:39.824047 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:09:39.824144 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:39.824113 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:39.824208 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:39.824186 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:09:40.016750 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:40.016680 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 01:04:38 +0000 UTC" deadline="2028-04-03 03:01:42.531451268 +0000 UTC" Nov 04 01:09:40.016750 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:40.016744 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12385h52m2.514711527s" Nov 04 01:09:41.606235 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:41.605840 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:41.606235 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:41.605912 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs\") pod \"network-metrics-daemon-zfj7b\" (UID: \"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8\") " pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:41.606235 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:41.606048 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:09:41.606235 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:41.606104 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs podName:fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8 nodeName:}" failed. No retries permitted until 2026-11-04 01:09:45.606085958 +0000 UTC m=+10.282513607 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs") pod "network-metrics-daemon-zfj7b" (UID: "fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:09:41.606235 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:41.606169 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 01:09:41.606235 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:41.606205 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret podName:3ae06262-6e59-4775-bedc-5305ace1fae6 nodeName:}" failed. No retries permitted until 2026-11-04 01:09:45.606193105 +0000 UTC m=+10.282620748 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret") pod "global-pull-secret-syncer-b2jgm" (UID: "3ae06262-6e59-4775-bedc-5305ace1fae6") : object "kube-system"/"original-pull-secret" not registered Nov 04 01:09:41.707285 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:41.707225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxlfx\" (UniqueName: \"kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx\") pod \"network-check-target-xqtpx\" (UID: \"42f02121-70f2-43d1-be9d-2a81425ec97d\") " pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:41.707463 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:41.707422 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 01:09:41.707463 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:41.707443 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 01:09:41.707463 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:41.707456 2580 projected.go:196] Error preparing data for projected volume kube-api-access-sxlfx for pod openshift-network-diagnostics/network-check-target-xqtpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:09:41.707623 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:41.707533 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx podName:42f02121-70f2-43d1-be9d-2a81425ec97d nodeName:}" failed. No retries permitted until 2026-11-04 01:09:45.707512055 +0000 UTC m=+10.383939706 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sxlfx" (UniqueName: "kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx") pod "network-check-target-xqtpx" (UID: "42f02121-70f2-43d1-be9d-2a81425ec97d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:09:41.823683 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:41.823650 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:41.823865 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:41.823787 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:09:41.823865 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:41.823857 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:41.823986 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:41.823943 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:09:41.824049 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:41.823988 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:41.824097 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:41.824059 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:09:42.870770 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:42.869441 2580 generic.go:356] "Generic (PLEG): container finished" podID="825c4932c3373ac03b157d52d106cbb9" containerID="380876e5dd3f8ead4729c25a92baf91b3c52503c3326f032a866851288805b17" exitCode=0 Nov 04 01:09:42.870770 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:42.869776 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" event={"ID":"825c4932c3373ac03b157d52d106cbb9","Type":"ContainerDied","Data":"380876e5dd3f8ead4729c25a92baf91b3c52503c3326f032a866851288805b17"} Nov 04 01:09:42.876531 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:42.876246 2580 generic.go:356] "Generic (PLEG): container finished" podID="c3d08d00-e371-4793-955e-c5b51eef7879" containerID="9064fb38fafb124af725ba06fa575206f4950d82090bb2e06815146a9d61fce9" exitCode=0 Nov 04 01:09:42.876531 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:42.876288 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dg8wm" event={"ID":"c3d08d00-e371-4793-955e-c5b51eef7879","Type":"ContainerDied","Data":"9064fb38fafb124af725ba06fa575206f4950d82090bb2e06815146a9d61fce9"} Nov 04 01:09:43.823666 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:43.823589 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:43.823666 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:43.823624 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:43.823666 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:43.823624 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:43.824014 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:43.823725 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:09:43.824014 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:43.823808 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:09:43.824014 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:43.823908 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:09:43.883253 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:43.883215 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" event={"ID":"825c4932c3373ac03b157d52d106cbb9","Type":"ContainerStarted","Data":"1fd3854760d63fc9afe23eb5fd429ff393881de388059fd28eead474b7545665"} Nov 04 01:09:43.889791 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:43.889758 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" event={"ID":"5be03f35a536944b4cf2058a360cdcb6","Type":"ContainerStarted","Data":"a5b67596bec2328d8975cc1759f5de4cba7538ae734f75497915ec10209e222d"} Nov 04 01:09:43.894433 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:43.894390 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qvjtd" event={"ID":"a06edce6-baf3-40bc-9d0e-a6e71c393165","Type":"ContainerStarted","Data":"5760dcaebddb50f21224d4f08a11a1e6cad80d395773a3518dfa836403202c23"} Nov 04 01:09:44.012869 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:44.012805 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" podStartSLOduration=7.01278454 podStartE2EDuration="7.01278454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:44.012163952 +0000 UTC m=+8.688591606" watchObservedRunningTime="2026-11-04 01:09:44.01278454 +0000 UTC m=+8.689212193" Nov 04 01:09:44.268654 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:44.268598 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qvjtd" podStartSLOduration=3.174730148 podStartE2EDuration="8.268576957s" totalImagesPullingTime="5.093846809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:44.139604264 +0000 UTC m=+8.816032121" watchObservedRunningTime="2026-11-04 01:09:44.268576957 +0000 UTC m=+8.945004610" Nov 04 01:09:44.268829 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:44.268711 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" podStartSLOduration=7.268704421 podStartE2EDuration="7.268704421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:44.267867273 +0000 UTC m=+8.944294924" watchObservedRunningTime="2026-11-04 01:09:44.268704421 +0000 UTC m=+8.945132073" Nov 04 01:09:44.902143 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:44.902075 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zzx84" event={"ID":"0f50bc9b-91da-4d92-8553-c9f9476665e4","Type":"ContainerStarted","Data":"ffea712e9e89101ebcd2851101b9717e202c0f2ce493fa284d918dd1f20c80dc"} Nov 04 01:09:45.555083 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:45.555049 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qvjtd" Nov 04 01:09:45.555589 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:45.555561 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qvjtd" Nov 04 01:09:45.643153 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:45.643120 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs\") pod \"network-metrics-daemon-zfj7b\" (UID: \"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8\") " pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:45.643314 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:45.643191 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:45.643370 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:45.643316 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 01:09:45.643455 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:45.643375 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret podName:3ae06262-6e59-4775-bedc-5305ace1fae6 nodeName:}" failed. No retries permitted until 2026-11-04 01:09:53.643354888 +0000 UTC m=+18.319782536 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret") pod "global-pull-secret-syncer-b2jgm" (UID: "3ae06262-6e59-4775-bedc-5305ace1fae6") : object "kube-system"/"original-pull-secret" not registered Nov 04 01:09:45.643455 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:45.643449 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:09:45.643572 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:45.643483 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs podName:fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8 nodeName:}" failed. No retries permitted until 2026-11-04 01:09:53.643471507 +0000 UTC m=+18.319899148 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs") pod "network-metrics-daemon-zfj7b" (UID: "fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:09:45.672629 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:45.672575 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zzx84" podStartSLOduration=3.603956401 podStartE2EDuration="9.672558325s" totalImagesPullingTime="6.068601924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:45.030964968 +0000 UTC m=+9.707392622" watchObservedRunningTime="2026-11-04 01:09:45.672558325 +0000 UTC m=+10.348985972" Nov 04 01:09:45.744044 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:45.743973 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxlfx\" (UniqueName: \"kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx\") pod \"network-check-target-xqtpx\" (UID: \"42f02121-70f2-43d1-be9d-2a81425ec97d\") " pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:45.744216 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:45.744173 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 01:09:45.744216 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:45.744195 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 01:09:45.744216 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:45.744206 2580 projected.go:196] Error preparing data for projected volume kube-api-access-sxlfx for pod openshift-network-diagnostics/network-check-target-xqtpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:09:45.744375 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:45.744256 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx podName:42f02121-70f2-43d1-be9d-2a81425ec97d nodeName:}" failed. No retries permitted until 2026-11-04 01:09:53.744240084 +0000 UTC m=+18.420667723 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sxlfx" (UniqueName: "kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx") pod "network-check-target-xqtpx" (UID: "42f02121-70f2-43d1-be9d-2a81425ec97d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:09:45.823853 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:45.823772 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:45.823988 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:45.823900 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:09:45.824870 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:45.824844 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:45.824988 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:45.824948 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:09:45.826625 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:45.826355 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:45.826625 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:45.826481 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:09:45.907352 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:45.907316 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" event={"ID":"82966365-6bec-467d-9279-12ac45dd01e2","Type":"ContainerStarted","Data":"c705c6be308edc6f8b32e06e2b2aa6fa105b25e6e82f83d8197fb004779faf55"} Nov 04 01:09:46.908800 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:46.908733 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 01:09:47.454330 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:47.454278 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 01:09:47.819746 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:47.819583 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T01:09:47.454294648Z","UUID":"fb0a6044-149e-4284-b197-8118ad804beb","Handler":null,"Name":"","Endpoint":""} Nov 04 01:09:47.820680 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:47.820508 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:47.820812 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:47.820525 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:47.820812 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:47.820773 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:09:47.820893 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:47.820831 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:09:47.820893 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:47.820563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:47.820968 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:47.820909 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:09:47.821432 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:47.821393 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 01:09:47.821432 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:47.821433 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 01:09:47.911641 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:47.911608 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zn87s" event={"ID":"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5","Type":"ContainerStarted","Data":"3759fcc7a623bdb9737ef8f27f52f9ae89ee874fa41b25f9d9f4ae62416e3f7f"} Nov 04 01:09:47.913576 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:47.913548 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" event={"ID":"82966365-6bec-467d-9279-12ac45dd01e2","Type":"ContainerStarted","Data":"ec7f808cb6753e208824588f4eabf08b99dbee102e105fbb952a3cae3f2cfc27"} Nov 04 01:09:47.915090 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:47.915045 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xsmlx" event={"ID":"ffb55ed6-b225-4db5-a7fb-66a10868befc","Type":"ContainerStarted","Data":"06c707083dda03fb13a38d23cea0c9ab8b02138945c845abc3ecd410487f2218"} Nov 04 01:09:48.031861 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:48.031801 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-zn87s" podStartSLOduration=2.320682382 podStartE2EDuration="11.031784766s" totalImagesPullingTime="8.711102384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:48.031372036 +0000 UTC m=+12.707799725" watchObservedRunningTime="2026-11-04 01:09:48.031784766 +0000 UTC m=+12.708212416" Nov 04 01:09:48.150273 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:48.150199 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-xsmlx" podStartSLOduration=3.078039736 podStartE2EDuration="12.150187268s" totalImagesPullingTime="9.072147532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:48.149755458 +0000 UTC m=+12.826183105" watchObservedRunningTime="2026-11-04 01:09:48.150187268 +0000 UTC m=+12.826614915" Nov 04 01:09:48.918736 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:48.918702 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" event={"ID":"03f3058b-0663-48a7-ab67-a41bfe5a2c57","Type":"ContainerStarted","Data":"9fbdb3650ea7fc1ea4811ac3a78122aa20ab832ee4cfb2c61299e82e8e433e8c"} Nov 04 01:09:48.923315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:48.922925 2580 generic.go:356] "Generic (PLEG): container finished" podID="c3d08d00-e371-4793-955e-c5b51eef7879" containerID="d174cba8a906028140ba5b2f6b494efc339d6f8290cc98b395e007b50c2ef9f1" exitCode=0 Nov 04 01:09:48.923315 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:48.923174 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dg8wm" event={"ID":"c3d08d00-e371-4793-955e-c5b51eef7879","Type":"ContainerDied","Data":"d174cba8a906028140ba5b2f6b494efc339d6f8290cc98b395e007b50c2ef9f1"} Nov 04 01:09:48.925406 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:48.925384 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rmvz5" event={"ID":"c5b0a3f1-5189-4572-9765-20eb3960fbae","Type":"ContainerStarted","Data":"eb41d4499f6bfc36cd6a737627ebded87411b946bc859229f94dd6bea58b2a8e"} Nov 04 01:09:49.039957 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:49.039908 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" podStartSLOduration=1.513128915 podStartE2EDuration="12.039896064s" totalImagesPullingTime="10.526767149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:49.039833332 +0000 UTC m=+13.716260978" watchObservedRunningTime="2026-11-04 01:09:49.039896064 +0000 UTC m=+13.716323710" Nov 04 01:09:49.279187 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:49.279131 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rmvz5" podStartSLOduration=4.383761822 podStartE2EDuration="13.279113911s" totalImagesPullingTime="8.895352089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:49.278518002 +0000 UTC m=+13.954945649" watchObservedRunningTime="2026-11-04 01:09:49.279113911 +0000 UTC m=+13.955541560" Nov 04 01:09:49.820998 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:49.820925 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:49.820998 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:49.820966 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:49.821166 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:49.820933 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:49.821166 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:49.821034 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:09:49.821166 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:49.821108 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:09:49.821317 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:49.821195 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:09:49.929679 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:49.929646 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" event={"ID":"82966365-6bec-467d-9279-12ac45dd01e2","Type":"ContainerStarted","Data":"11292157da0f3715e76170a07d8316300627d46520612e4955fb8db494173a58"} Nov 04 01:09:50.049232 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:50.049181 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" podStartSLOduration=3.08045278 podStartE2EDuration="14.049165332s" totalImagesPullingTime="10.968712552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:09:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:50.049072437 +0000 UTC m=+14.725500083" watchObservedRunningTime="2026-11-04 01:09:50.049165332 +0000 UTC m=+14.725592977" Nov 04 01:09:50.244235 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:50.244210 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qvjtd" Nov 04 01:09:50.244377 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:50.244365 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 01:09:50.244744 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:50.244718 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qvjtd" Nov 04 01:09:50.933201 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:50.933121 2580 generic.go:356] "Generic (PLEG): container finished" podID="c3d08d00-e371-4793-955e-c5b51eef7879" containerID="69d11914966474f22c8f724cb3564599307f7405d0d239dbdbf2810fd734636d" exitCode=0 Nov 04 01:09:50.933657 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:50.933242 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dg8wm" event={"ID":"c3d08d00-e371-4793-955e-c5b51eef7879","Type":"ContainerDied","Data":"69d11914966474f22c8f724cb3564599307f7405d0d239dbdbf2810fd734636d"} Nov 04 01:09:51.820930 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:51.820758 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:51.821072 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:51.820809 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:51.821150 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:51.820831 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:51.821224 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:51.821210 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:09:51.821269 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:51.821129 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:09:51.821269 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:51.821001 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:09:52.939034 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:52.939000 2580 generic.go:356] "Generic (PLEG): container finished" podID="c3d08d00-e371-4793-955e-c5b51eef7879" containerID="2073dc217017c30a1734dddbfb01109307805494a8f6e42b0a4fa6584709a574" exitCode=0 Nov 04 01:09:52.939376 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:52.939049 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dg8wm" event={"ID":"c3d08d00-e371-4793-955e-c5b51eef7879","Type":"ContainerDied","Data":"2073dc217017c30a1734dddbfb01109307805494a8f6e42b0a4fa6584709a574"} Nov 04 01:09:53.707965 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:53.707934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:53.708143 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:53.707985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs\") pod \"network-metrics-daemon-zfj7b\" (UID: \"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8\") " pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:53.708143 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:53.708073 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:09:53.708143 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:53.708073 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 01:09:53.708143 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:53.708117 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs podName:fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:09.708105162 +0000 UTC m=+34.384532800 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs") pod "network-metrics-daemon-zfj7b" (UID: "fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:09:53.708143 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:53.708128 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret podName:3ae06262-6e59-4775-bedc-5305ace1fae6 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:09.708123178 +0000 UTC m=+34.384550803 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret") pod "global-pull-secret-syncer-b2jgm" (UID: "3ae06262-6e59-4775-bedc-5305ace1fae6") : object "kube-system"/"original-pull-secret" not registered Nov 04 01:09:53.809273 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:53.809241 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxlfx\" (UniqueName: \"kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx\") pod \"network-check-target-xqtpx\" (UID: \"42f02121-70f2-43d1-be9d-2a81425ec97d\") " pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:53.809392 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:53.809349 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 01:09:53.809392 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:53.809362 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 01:09:53.809392 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:53.809370 2580 projected.go:196] Error preparing data for projected volume kube-api-access-sxlfx for pod openshift-network-diagnostics/network-check-target-xqtpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:09:53.809504 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:53.809425 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx podName:42f02121-70f2-43d1-be9d-2a81425ec97d nodeName:}" failed. No retries permitted until 2026-11-04 01:10:09.809398496 +0000 UTC m=+34.485826135 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sxlfx" (UniqueName: "kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx") pod "network-check-target-xqtpx" (UID: "42f02121-70f2-43d1-be9d-2a81425ec97d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:09:53.821440 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:53.821404 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:53.821548 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:53.821451 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:53.821548 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:53.821525 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:09:53.821631 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:53.821600 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:09:53.821663 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:53.821645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:53.821783 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:53.821762 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:09:55.822151 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:55.822120 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:55.822658 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:55.822280 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:09:55.822779 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:55.822758 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:55.822924 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:55.822907 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:09:55.823083 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:55.823032 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:55.823471 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:55.823429 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:09:57.820806 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:57.820770 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:57.821254 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:57.820779 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:57.821254 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:57.820899 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:09:57.821254 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:57.820910 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:57.821254 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:57.820984 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:09:57.821254 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:57.821056 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:09:57.951304 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:57.951271 2580 generic.go:356] "Generic (PLEG): container finished" podID="c3d08d00-e371-4793-955e-c5b51eef7879" containerID="f22b984b447dcd40a855acc1ef8783116a6005d7e31a639ac5257b4d1bcd5bc7" exitCode=0 Nov 04 01:09:57.951461 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:57.951309 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dg8wm" event={"ID":"c3d08d00-e371-4793-955e-c5b51eef7879","Type":"ContainerDied","Data":"f22b984b447dcd40a855acc1ef8783116a6005d7e31a639ac5257b4d1bcd5bc7"} Nov 04 01:09:58.956772 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:58.956740 2580 generic.go:356] "Generic (PLEG): container finished" podID="c3d08d00-e371-4793-955e-c5b51eef7879" containerID="1985d867be774208da17fe985b40d74948d4d2c980ca8ede515f7cfbb987339a" exitCode=0 Nov 04 01:09:58.957224 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:58.956778 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dg8wm" event={"ID":"c3d08d00-e371-4793-955e-c5b51eef7879","Type":"ContainerDied","Data":"1985d867be774208da17fe985b40d74948d4d2c980ca8ede515f7cfbb987339a"} Nov 04 01:09:59.821568 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:59.821380 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:09:59.821724 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:59.821410 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:09:59.821724 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:59.821650 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:09:59.821724 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:59.821468 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:09:59.821724 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:59.821709 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:09:59.821876 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:09:59.821770 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:09:59.962085 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:59.961922 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" event={"ID":"89bd09a4-6d44-49ae-9bfa-837e2e5bf915","Type":"ContainerStarted","Data":"5dcf82758e62b72d09bdbb870c956f55a055cc5e801490469660881270b13679"} Nov 04 01:09:59.962085 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:59.961981 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" event={"ID":"89bd09a4-6d44-49ae-9bfa-837e2e5bf915","Type":"ContainerStarted","Data":"682ddc9d19f097df61dc468422d22ef7be9e88543268707cf77439d13cce5fcb"} Nov 04 01:09:59.962085 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:59.962011 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" event={"ID":"89bd09a4-6d44-49ae-9bfa-837e2e5bf915","Type":"ContainerStarted","Data":"db97f2e57e979433209ccae1433e3d542e4668bca80e90f4d2d8cf952f701db0"} Nov 04 01:09:59.962085 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:59.962047 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" event={"ID":"89bd09a4-6d44-49ae-9bfa-837e2e5bf915","Type":"ContainerStarted","Data":"4ddad9c607a2f11b92da5537b92f5ace27e50b28e773fcac81fd7592481800b2"} Nov 04 01:09:59.962085 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:59.962075 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" event={"ID":"89bd09a4-6d44-49ae-9bfa-837e2e5bf915","Type":"ContainerStarted","Data":"53b3e96b2dcd1f9a2c8fbf5214716b082dc9d3fbacd76ff86bee2d6b65b9fd51"} Nov 04 01:09:59.962826 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:59.962099 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" event={"ID":"89bd09a4-6d44-49ae-9bfa-837e2e5bf915","Type":"ContainerStarted","Data":"da7e908b08ef6f938585bde30459c212771a4f41bf5fee7469d19aab57a0f13f"} Nov 04 01:09:59.965101 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:09:59.965078 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dg8wm" event={"ID":"c3d08d00-e371-4793-955e-c5b51eef7879","Type":"ContainerStarted","Data":"b927b650dfb1fc38023bfadf4a18580413154212ce18beb37d74b0529fd39b61"} Nov 04 01:10:00.088349 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:00.088284 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dg8wm" podStartSLOduration=7.5242807670000005 podStartE2EDuration="24.088273993s" totalImagesPullingTime="16.563993226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:09:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:00.088196571 +0000 UTC m=+24.764624217" watchObservedRunningTime="2026-11-04 01:10:00.088273993 +0000 UTC m=+24.764701640" Nov 04 01:10:01.821398 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:01.821363 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:10:01.821774 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:01.821428 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:10:01.821774 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:01.821449 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:01.821774 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:01.821528 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:10:01.821774 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:01.821617 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:10:01.821774 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:01.821703 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:10:01.972566 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:01.972538 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" event={"ID":"89bd09a4-6d44-49ae-9bfa-837e2e5bf915","Type":"ContainerStarted","Data":"6a52d436c26cad50e3d18070c70addd0e153e53789d8a9251ca319819aff277f"} Nov 04 01:10:03.821367 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:03.821341 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:10:03.821994 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:03.821368 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:03.821994 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:03.821476 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:10:03.821994 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:03.821486 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:10:03.821994 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:03.821550 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:10:03.821994 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:03.821624 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:10:03.979384 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:03.979359 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" event={"ID":"89bd09a4-6d44-49ae-9bfa-837e2e5bf915","Type":"ContainerStarted","Data":"6e1383c0f9ee61107a80dd10023bb7663ccbc6597b62deb7aff8a499ba7bed6c"} Nov 04 01:10:03.979712 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:03.979685 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:03.979832 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:03.979759 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:03.979832 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:03.979809 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:03.995379 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:03.995357 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:03.995538 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:03.995504 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:04.261270 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:04.261231 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" podStartSLOduration=7.543970897 podStartE2EDuration="28.261218421s" totalImagesPullingTime="20.717247524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:04.260996132 +0000 UTC m=+28.937423778" watchObservedRunningTime="2026-11-04 01:10:04.261218421 +0000 UTC m=+28.937646067" Nov 04 01:10:05.821424 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:05.821395 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:10:05.821909 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:05.821523 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:10:05.821909 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:05.821546 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:10:05.821909 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:05.821574 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:05.821909 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:05.821639 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:10:05.821909 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:05.821704 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:10:06.034086 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:06.034049 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b2jgm"] Nov 04 01:10:06.034255 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:06.034182 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:06.034298 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:06.034274 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:10:06.043022 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:06.042999 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zfj7b"] Nov 04 01:10:06.043244 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:06.043168 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:10:06.043295 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:06.043251 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:10:06.045290 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:06.045268 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xqtpx"] Nov 04 01:10:06.045524 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:06.045500 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:10:06.045609 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:06.045568 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:10:07.821214 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:07.821134 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:10:07.821214 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:07.821134 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:10:07.821679 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:07.821259 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:07.821679 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:07.821265 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:10:07.821679 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:07.821351 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:10:07.821679 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:07.821469 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:10:09.730937 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:09.730903 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs\") pod \"network-metrics-daemon-zfj7b\" (UID: \"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8\") " pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:10:09.731284 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:09.730948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:09.731284 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:09.731031 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 01:10:09.731284 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:09.731034 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:10:09.731284 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:09.731077 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret podName:3ae06262-6e59-4775-bedc-5305ace1fae6 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:41.731065825 +0000 UTC m=+66.407493451 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret") pod "global-pull-secret-syncer-b2jgm" (UID: "3ae06262-6e59-4775-bedc-5305ace1fae6") : object "kube-system"/"original-pull-secret" not registered Nov 04 01:10:09.731284 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:09.731091 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs podName:fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:41.731084625 +0000 UTC m=+66.407512251 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs") pod "network-metrics-daemon-zfj7b" (UID: "fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 01:10:09.821257 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:09.821235 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:10:09.821257 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:09.821247 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:10:09.821446 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:09.821241 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:09.821446 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:09.821324 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:10:09.821446 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:09.821400 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:10:09.821570 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:09.821488 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:10:09.831921 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:09.831900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxlfx\" (UniqueName: \"kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx\") pod \"network-check-target-xqtpx\" (UID: \"42f02121-70f2-43d1-be9d-2a81425ec97d\") " pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:10:09.832027 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:09.831996 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 01:10:09.832027 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:09.832008 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 01:10:09.832027 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:09.832016 2580 projected.go:196] Error preparing data for projected volume kube-api-access-sxlfx for pod openshift-network-diagnostics/network-check-target-xqtpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:10:09.832115 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:09.832053 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx podName:42f02121-70f2-43d1-be9d-2a81425ec97d nodeName:}" failed. No retries permitted until 2026-11-04 01:10:41.832042936 +0000 UTC m=+66.508470574 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sxlfx" (UniqueName: "kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx") pod "network-check-target-xqtpx" (UID: "42f02121-70f2-43d1-be9d-2a81425ec97d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 01:10:11.821317 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:11.821251 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:10:11.821317 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:11.821309 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:10:11.821733 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:11.821317 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:11.821733 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:11.821435 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfj7b" podUID="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" Nov 04 01:10:11.821733 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:11.821507 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqtpx" podUID="42f02121-70f2-43d1-be9d-2a81425ec97d" Nov 04 01:10:11.821733 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:11.821617 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b2jgm" podUID="3ae06262-6e59-4775-bedc-5305ace1fae6" Nov 04 01:10:12.100442 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.100382 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeReady" Nov 04 01:10:12.100544 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.100491 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 04 01:10:12.217047 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.217028 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-65nzk"] Nov 04 01:10:12.298024 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.298002 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-d26rm"] Nov 04 01:10:12.298125 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.298106 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-65nzk" Nov 04 01:10:12.310705 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.310680 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xzhqb"] Nov 04 01:10:12.311001 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.310973 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-d26rm" Nov 04 01:10:12.332844 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.332822 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl"] Nov 04 01:10:12.332958 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.332942 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xzhqb" Nov 04 01:10:12.346220 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.346199 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-dgdxz"] Nov 04 01:10:12.346310 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.346289 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:12.360476 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.360455 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-f5c95ffcf-hl6wf"] Nov 04 01:10:12.360559 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.360550 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.362445 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.362410 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 04 01:10:12.362542 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.362530 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 04 01:10:12.384669 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.384650 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf"] Nov 04 01:10:12.385386 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.385365 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:12.398625 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.398608 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz"] Nov 04 01:10:12.398728 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.398714 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" Nov 04 01:10:12.417075 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.416905 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 04 01:10:12.417075 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.416928 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9qr5d\"" Nov 04 01:10:12.417075 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.416930 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9sllv\"" Nov 04 01:10:12.421532 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.421514 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 04 01:10:12.422832 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.422818 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2w74l"] Nov 04 01:10:12.422929 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.422915 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" Nov 04 01:10:12.450514 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.450491 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k"] Nov 04 01:10:12.450599 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.450587 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2w74l" Nov 04 01:10:12.450665 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.450601 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cecaa47-44ef-440e-80eb-45230fe3edc0-service-ca-bundle\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.450665 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.450633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tf5p\" (UniqueName: \"kubernetes.io/projected/cd605a92-f88c-40ea-bde7-53c2da66a576-kube-api-access-9tf5p\") pod \"kas-connection-checker-685895c7c9-d26rm\" (UID: \"cd605a92-f88c-40ea-bde7-53c2da66a576\") " pod="kube-system/kas-connection-checker-685895c7c9-d26rm" Nov 04 01:10:12.450761 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.450661 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm4k6\" (UniqueName: \"kubernetes.io/projected/75c5b017-8f43-41dc-b79b-0353de290289-kube-api-access-dm4k6\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:12.450761 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.450750 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpfww\" (UniqueName: \"kubernetes.io/projected/a23c835a-aacb-4f92-a2cd-799f5e870b85-kube-api-access-kpfww\") pod \"kas-connection-checker-685895c7c9-65nzk\" (UID: \"a23c835a-aacb-4f92-a2cd-799f5e870b85\") " pod="kube-system/kas-connection-checker-685895c7c9-65nzk" Nov 04 01:10:12.450861 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.450796 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/75c5b017-8f43-41dc-b79b-0353de290289-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:12.450913 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.450856 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cecaa47-44ef-440e-80eb-45230fe3edc0-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.450983 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.450963 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dxcj\" (UniqueName: \"kubernetes.io/projected/a8b2b155-18ef-40a1-b87b-a6e44e130d65-kube-api-access-9dxcj\") pod \"kas-connection-checker-685895c7c9-xzhqb\" (UID: \"a8b2b155-18ef-40a1-b87b-a6e44e130d65\") " pod="kube-system/kas-connection-checker-685895c7c9-xzhqb" Nov 04 01:10:12.451026 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.450994 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cecaa47-44ef-440e-80eb-45230fe3edc0-tmp\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.451099 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.451084 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d94r5\" (UniqueName: \"kubernetes.io/projected/5cecaa47-44ef-440e-80eb-45230fe3edc0-kube-api-access-d94r5\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.451144 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.451131 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5cecaa47-44ef-440e-80eb-45230fe3edc0-snapshots\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.451184 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.451157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cecaa47-44ef-440e-80eb-45230fe3edc0-serving-cert\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.451229 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.451182 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:12.463197 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.463173 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 04 01:10:12.463330 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.463205 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 01:10:12.463330 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.463221 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 01:10:12.463330 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.463249 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 01:10:12.463330 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.463293 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 01:10:12.466616 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.466601 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-8cfg7"] Nov 04 01:10:12.466689 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.466678 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:12.485657 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.485637 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-99c5d585-rzfqd"] Nov 04 01:10:12.485752 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.485737 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:12.501713 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.501697 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-582gz"] Nov 04 01:10:12.501817 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.501803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.514023 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.514004 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 04 01:10:12.514127 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.514075 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 04 01:10:12.514210 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.514162 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:10:12.514210 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.514175 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 04 01:10:12.514210 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.514166 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 01:10:12.514297 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.514163 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 04 01:10:12.514297 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.514220 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 01:10:12.514297 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.514244 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 04 01:10:12.514297 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.514255 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 04 01:10:12.514297 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.514259 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:10:12.514297 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.514248 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 04 01:10:12.514841 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.514829 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 04 01:10:12.515002 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.514991 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:10:12.517053 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.517035 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 04 01:10:12.517149 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.517050 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 04 01:10:12.517149 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.517091 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 04 01:10:12.517149 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.517115 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tjkgs\"" Nov 04 01:10:12.517322 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.517306 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-qmn7f\"" Nov 04 01:10:12.517361 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.517339 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 04 01:10:12.517361 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.517344 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ltqgp\"" Nov 04 01:10:12.517361 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.517348 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-f6ftn\"" Nov 04 01:10:12.517465 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.517365 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 04 01:10:12.520080 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.520065 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:10:12.520135 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.520092 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 04 01:10:12.520135 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.520097 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ngfvk\"" Nov 04 01:10:12.520135 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.520104 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 04 01:10:12.520135 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.520123 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 04 01:10:12.520977 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.520960 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6556fd5cf6-2dl7j"] Nov 04 01:10:12.521065 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.521054 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" Nov 04 01:10:12.538496 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.538479 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 04 01:10:12.545132 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.545112 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-dbnq4"] Nov 04 01:10:12.545223 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.545211 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.551702 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.551683 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d94r5\" (UniqueName: \"kubernetes.io/projected/5cecaa47-44ef-440e-80eb-45230fe3edc0-kube-api-access-d94r5\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.551784 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.551718 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-default-certificate\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:12.551784 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.551767 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5cecaa47-44ef-440e-80eb-45230fe3edc0-snapshots\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.551849 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.551796 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cecaa47-44ef-440e-80eb-45230fe3edc0-serving-cert\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.551849 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.551824 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:12.551906 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.551850 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7mxs\" (UniqueName: \"kubernetes.io/projected/a7991c77-7106-4c73-8812-ff6d4c06b666-kube-api-access-b7mxs\") pod \"volume-data-source-validator-58c9f5f46d-2w74l\" (UID: \"a7991c77-7106-4c73-8812-ff6d4c06b666\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2w74l" Nov 04 01:10:12.551906 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.551878 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:12.551967 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.551911 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7573b3c8-fd6e-41c5-9de4-9c441f2ce026-serving-cert\") pod \"service-ca-operator-544d749c8c-d2dsz\" (UID: \"7573b3c8-fd6e-41c5-9de4-9c441f2ce026\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" Nov 04 01:10:12.551967 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.551929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b44a2a16-b00a-4113-b0ab-5cdcb955b325-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-29mpf\" (UID: \"b44a2a16-b00a-4113-b0ab-5cdcb955b325\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" Nov 04 01:10:12.552022 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.551968 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b44a2a16-b00a-4113-b0ab-5cdcb955b325-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-29mpf\" (UID: \"b44a2a16-b00a-4113-b0ab-5cdcb955b325\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" Nov 04 01:10:12.552022 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.551992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pbbl\" (UniqueName: \"kubernetes.io/projected/b44a2a16-b00a-4113-b0ab-5cdcb955b325-kube-api-access-2pbbl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-29mpf\" (UID: \"b44a2a16-b00a-4113-b0ab-5cdcb955b325\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" Nov 04 01:10:12.552022 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.552010 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7573b3c8-fd6e-41c5-9de4-9c441f2ce026-config\") pod \"service-ca-operator-544d749c8c-d2dsz\" (UID: \"7573b3c8-fd6e-41c5-9de4-9c441f2ce026\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" Nov 04 01:10:12.552022 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:12.552015 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 01:10:12.552133 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.552028 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:12.552133 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:12.552070 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls podName:75c5b017-8f43-41dc-b79b-0353de290289 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:13.052050735 +0000 UTC m=+37.728478373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fh4kl" (UID: "75c5b017-8f43-41dc-b79b-0353de290289") : secret "cluster-monitoring-operator-tls" not found Nov 04 01:10:12.552133 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.552091 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsq8f\" (UniqueName: \"kubernetes.io/projected/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-kube-api-access-gsq8f\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:12.552219 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.552139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cecaa47-44ef-440e-80eb-45230fe3edc0-service-ca-bundle\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.552247 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.552218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tf5p\" (UniqueName: \"kubernetes.io/projected/cd605a92-f88c-40ea-bde7-53c2da66a576-kube-api-access-9tf5p\") pod \"kas-connection-checker-685895c7c9-d26rm\" (UID: \"cd605a92-f88c-40ea-bde7-53c2da66a576\") " pod="kube-system/kas-connection-checker-685895c7c9-d26rm" Nov 04 01:10:12.552276 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.552244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm4k6\" (UniqueName: \"kubernetes.io/projected/75c5b017-8f43-41dc-b79b-0353de290289-kube-api-access-dm4k6\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:12.552316 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.552277 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-stats-auth\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:12.552424 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.552394 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5cecaa47-44ef-440e-80eb-45230fe3edc0-snapshots\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.552487 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.552398 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:12.552487 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.552460 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp5d9\" (UniqueName: \"kubernetes.io/projected/7573b3c8-fd6e-41c5-9de4-9c441f2ce026-kube-api-access-dp5d9\") pod \"service-ca-operator-544d749c8c-d2dsz\" (UID: \"7573b3c8-fd6e-41c5-9de4-9c441f2ce026\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" Nov 04 01:10:12.552564 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.552499 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpfww\" (UniqueName: \"kubernetes.io/projected/a23c835a-aacb-4f92-a2cd-799f5e870b85-kube-api-access-kpfww\") pod \"kas-connection-checker-685895c7c9-65nzk\" (UID: \"a23c835a-aacb-4f92-a2cd-799f5e870b85\") " pod="kube-system/kas-connection-checker-685895c7c9-65nzk" Nov 04 01:10:12.552564 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.552542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/75c5b017-8f43-41dc-b79b-0353de290289-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:12.552649 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.552574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:12.552649 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.552598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl9gz\" (UniqueName: \"kubernetes.io/projected/f2df8dc5-6403-4827-b6d6-9c209c595e05-kube-api-access-jl9gz\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:12.552649 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.552615 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cecaa47-44ef-440e-80eb-45230fe3edc0-service-ca-bundle\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.552794 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.552689 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cecaa47-44ef-440e-80eb-45230fe3edc0-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.552794 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.552710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dxcj\" (UniqueName: \"kubernetes.io/projected/a8b2b155-18ef-40a1-b87b-a6e44e130d65-kube-api-access-9dxcj\") pod \"kas-connection-checker-685895c7c9-xzhqb\" (UID: \"a8b2b155-18ef-40a1-b87b-a6e44e130d65\") " pod="kube-system/kas-connection-checker-685895c7c9-xzhqb" Nov 04 01:10:12.552794 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.552724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cecaa47-44ef-440e-80eb-45230fe3edc0-tmp\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.553084 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.553061 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cecaa47-44ef-440e-80eb-45230fe3edc0-tmp\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.553209 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.553191 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/75c5b017-8f43-41dc-b79b-0353de290289-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:12.553394 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.553379 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cecaa47-44ef-440e-80eb-45230fe3edc0-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.555459 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.555444 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cecaa47-44ef-440e-80eb-45230fe3edc0-serving-cert\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.559324 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.559306 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2h8lq"] Nov 04 01:10:12.559451 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.559425 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dbnq4" Nov 04 01:10:12.566877 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.566859 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:10:12.566946 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.566876 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 04 01:10:12.566946 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.566875 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s46xs\"" Nov 04 01:10:12.567017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.566986 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 04 01:10:12.567017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.566995 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 04 01:10:12.567017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.567003 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-d62tf\"" Nov 04 01:10:12.567097 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.567083 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 04 01:10:12.567133 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.567093 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 04 01:10:12.568566 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.568553 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 04 01:10:12.568810 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.568797 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 04 01:10:12.584169 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.584148 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 04 01:10:12.592258 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.592243 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ms2pt"] Nov 04 01:10:12.592360 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.592348 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:12.613208 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.613168 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d94r5\" (UniqueName: \"kubernetes.io/projected/5cecaa47-44ef-440e-80eb-45230fe3edc0-kube-api-access-d94r5\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.615986 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.615964 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-d26rm"] Nov 04 01:10:12.616062 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.615996 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-65nzk"] Nov 04 01:10:12.616062 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.616016 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xzhqb"] Nov 04 01:10:12.616062 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.616034 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2w74l"] Nov 04 01:10:12.616062 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.616050 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf"] Nov 04 01:10:12.616230 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.616067 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k"] Nov 04 01:10:12.616230 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.616082 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-dgdxz"] Nov 04 01:10:12.616230 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.616093 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ms2pt" Nov 04 01:10:12.616230 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.616099 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f5c95ffcf-hl6wf"] Nov 04 01:10:12.616398 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.616286 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-dbnq4"] Nov 04 01:10:12.616398 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.616319 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2h8lq"] Nov 04 01:10:12.616398 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.616349 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl"] Nov 04 01:10:12.616398 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.616369 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6556fd5cf6-2dl7j"] Nov 04 01:10:12.616398 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.616395 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-99c5d585-rzfqd"] Nov 04 01:10:12.616647 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.616431 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz"] Nov 04 01:10:12.616647 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.616457 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ms2pt"] Nov 04 01:10:12.616647 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.616478 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-582gz"] Nov 04 01:10:12.616647 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.616500 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-8cfg7"] Nov 04 01:10:12.617212 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.617199 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 04 01:10:12.617297 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.617286 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-ltlc5\"" Nov 04 01:10:12.618334 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.618320 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-4dm6v\"" Nov 04 01:10:12.626607 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.626583 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dxcj\" (UniqueName: \"kubernetes.io/projected/a8b2b155-18ef-40a1-b87b-a6e44e130d65-kube-api-access-9dxcj\") pod \"kas-connection-checker-685895c7c9-xzhqb\" (UID: \"a8b2b155-18ef-40a1-b87b-a6e44e130d65\") " pod="kube-system/kas-connection-checker-685895c7c9-xzhqb" Nov 04 01:10:12.626703 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.626629 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tf5p\" (UniqueName: \"kubernetes.io/projected/cd605a92-f88c-40ea-bde7-53c2da66a576-kube-api-access-9tf5p\") pod \"kas-connection-checker-685895c7c9-d26rm\" (UID: \"cd605a92-f88c-40ea-bde7-53c2da66a576\") " pod="kube-system/kas-connection-checker-685895c7c9-d26rm" Nov 04 01:10:12.626761 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.626717 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpfww\" (UniqueName: \"kubernetes.io/projected/a23c835a-aacb-4f92-a2cd-799f5e870b85-kube-api-access-kpfww\") pod \"kas-connection-checker-685895c7c9-65nzk\" (UID: \"a23c835a-aacb-4f92-a2cd-799f5e870b85\") " pod="kube-system/kas-connection-checker-685895c7c9-65nzk" Nov 04 01:10:12.627087 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.627068 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm4k6\" (UniqueName: \"kubernetes.io/projected/75c5b017-8f43-41dc-b79b-0353de290289-kube-api-access-dm4k6\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:12.639996 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.639981 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-d26rm" Nov 04 01:10:12.653232 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653215 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b44a2a16-b00a-4113-b0ab-5cdcb955b325-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-29mpf\" (UID: \"b44a2a16-b00a-4113-b0ab-5cdcb955b325\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" Nov 04 01:10:12.653318 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653257 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae64edd1-43dc-406d-bcad-f7f34bb1f991-config\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:12.653318 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx7dg\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-kube-api-access-cx7dg\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.653318 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653291 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b44a2a16-b00a-4113-b0ab-5cdcb955b325-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-29mpf\" (UID: \"b44a2a16-b00a-4113-b0ab-5cdcb955b325\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" Nov 04 01:10:12.653318 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653307 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2aa236a-4ff4-4b39-90ff-c782721444ee-ca-trust-extracted\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.653476 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a99591bd-c354-4d6c-940d-51b0969ee916-image-registry-private-configuration\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.653514 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653483 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:12.653556 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653541 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsq8f\" (UniqueName: \"kubernetes.io/projected/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-kube-api-access-gsq8f\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:12.653605 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:12.653562 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 01:10:12.653605 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653576 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p78dk\" (UniqueName: \"kubernetes.io/projected/6250a398-c824-4c6e-9083-c7b0df543700-kube-api-access-p78dk\") pod \"network-check-source-6f78d59f99-dbnq4\" (UID: \"6250a398-c824-4c6e-9083-c7b0df543700\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dbnq4" Nov 04 01:10:12.653605 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-image-registry-private-configuration\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.653741 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:12.653620 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs podName:f2df8dc5-6403-4827-b6d6-9c209c595e05 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:13.153604355 +0000 UTC m=+37.830031997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs") pod "router-default-f5c95ffcf-hl6wf" (UID: "f2df8dc5-6403-4827-b6d6-9c209c595e05") : secret "router-metrics-certs-default" not found Nov 04 01:10:12.653741 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653639 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a99591bd-c354-4d6c-940d-51b0969ee916-trusted-ca\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.653741 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:12.653741 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653704 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-582gz\" (UID: \"5b8947a9-b2f0-4de2-a1b8-58342b6b3240\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" Nov 04 01:10:12.653741 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7573b3c8-fd6e-41c5-9de4-9c441f2ce026-config\") pod \"service-ca-operator-544d749c8c-d2dsz\" (UID: \"7573b3c8-fd6e-41c5-9de4-9c441f2ce026\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" Nov 04 01:10:12.653970 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653775 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v22h7\" (UniqueName: \"kubernetes.io/projected/ae64edd1-43dc-406d-bcad-f7f34bb1f991-kube-api-access-v22h7\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:12.653970 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:12.653815 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle podName:f2df8dc5-6403-4827-b6d6-9c209c595e05 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:13.153789742 +0000 UTC m=+37.830217390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle") pod "router-default-f5c95ffcf-hl6wf" (UID: "f2df8dc5-6403-4827-b6d6-9c209c595e05") : configmap references non-existent config key: service-ca.crt Nov 04 01:10:12.653970 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653835 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-nginx-conf\") pod \"networking-console-plugin-86c478bb88-582gz\" (UID: \"5b8947a9-b2f0-4de2-a1b8-58342b6b3240\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" Nov 04 01:10:12.653970 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653840 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b44a2a16-b00a-4113-b0ab-5cdcb955b325-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-29mpf\" (UID: \"b44a2a16-b00a-4113-b0ab-5cdcb955b325\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" Nov 04 01:10:12.653970 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653894 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a99591bd-c354-4d6c-940d-51b0969ee916-registry-certificates\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.653970 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653933 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-default-certificate\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:12.653970 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653951 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7mxs\" (UniqueName: \"kubernetes.io/projected/a7991c77-7106-4c73-8812-ff6d4c06b666-kube-api-access-b7mxs\") pod \"volume-data-source-validator-58c9f5f46d-2w74l\" (UID: \"a7991c77-7106-4c73-8812-ff6d4c06b666\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2w74l" Nov 04 01:10:12.653970 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653969 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5dhv\" (UniqueName: \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-kube-api-access-p5dhv\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.654277 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.653992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae64edd1-43dc-406d-bcad-f7f34bb1f991-trusted-ca\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:12.654277 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.654016 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a99591bd-c354-4d6c-940d-51b0969ee916-ca-trust-extracted\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.654277 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.654039 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pbbl\" (UniqueName: \"kubernetes.io/projected/b44a2a16-b00a-4113-b0ab-5cdcb955b325-kube-api-access-2pbbl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-29mpf\" (UID: \"b44a2a16-b00a-4113-b0ab-5cdcb955b325\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" Nov 04 01:10:12.654277 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.654078 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.654277 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.654217 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-bound-sa-token\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.654487 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.654276 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7573b3c8-fd6e-41c5-9de4-9c441f2ce026-config\") pod \"service-ca-operator-544d749c8c-d2dsz\" (UID: \"7573b3c8-fd6e-41c5-9de4-9c441f2ce026\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" Nov 04 01:10:12.654487 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.654284 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-stats-auth\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:12.654487 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.654326 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:12.654487 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.654348 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp5d9\" (UniqueName: \"kubernetes.io/projected/7573b3c8-fd6e-41c5-9de4-9c441f2ce026-kube-api-access-dp5d9\") pod \"service-ca-operator-544d749c8c-d2dsz\" (UID: \"7573b3c8-fd6e-41c5-9de4-9c441f2ce026\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" Nov 04 01:10:12.654487 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.654364 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae64edd1-43dc-406d-bcad-f7f34bb1f991-serving-cert\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:12.654487 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.654386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-certificates\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.654487 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:12.654430 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 01:10:12.654771 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:12.654490 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls podName:ef7aa61c-2789-49ed-989d-fa77fed0ff0f nodeName:}" failed. No retries permitted until 2026-11-04 01:10:13.154475346 +0000 UTC m=+37.830902989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q449k" (UID: "ef7aa61c-2789-49ed-989d-fa77fed0ff0f") : secret "samples-operator-tls" not found Nov 04 01:10:12.654771 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.654527 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl9gz\" (UniqueName: \"kubernetes.io/projected/f2df8dc5-6403-4827-b6d6-9c209c595e05-kube-api-access-jl9gz\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:12.654771 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.654615 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.654771 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.654644 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-installation-pull-secrets\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.654771 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.654670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-bound-sa-token\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.654771 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.654746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:12.654960 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.654813 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-trusted-ca\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.654960 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.654880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7573b3c8-fd6e-41c5-9de4-9c441f2ce026-serving-cert\") pod \"service-ca-operator-544d749c8c-d2dsz\" (UID: \"7573b3c8-fd6e-41c5-9de4-9c441f2ce026\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" Nov 04 01:10:12.654960 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.654911 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a99591bd-c354-4d6c-940d-51b0969ee916-installation-pull-secrets\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.655464 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.655441 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:12.655538 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.655461 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b44a2a16-b00a-4113-b0ab-5cdcb955b325-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-29mpf\" (UID: \"b44a2a16-b00a-4113-b0ab-5cdcb955b325\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" Nov 04 01:10:12.656288 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.656270 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-default-certificate\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:12.656523 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.656505 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-stats-auth\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:12.656594 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.656561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xzhqb" Nov 04 01:10:12.657078 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.657045 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7573b3c8-fd6e-41c5-9de4-9c441f2ce026-serving-cert\") pod \"service-ca-operator-544d749c8c-d2dsz\" (UID: \"7573b3c8-fd6e-41c5-9de4-9c441f2ce026\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" Nov 04 01:10:12.685081 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.685054 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 01:10:12.685081 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.685074 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 01:10:12.685226 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.685095 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 01:10:12.685226 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.685121 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 01:10:12.685226 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.685171 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 01:10:12.693453 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.693435 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:12.717902 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.717870 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 01:10:12.717995 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.717888 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ldpj7\"" Nov 04 01:10:12.717995 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.717975 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 01:10:12.718078 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.718005 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9dncz\"" Nov 04 01:10:12.729601 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.729574 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp5d9\" (UniqueName: \"kubernetes.io/projected/7573b3c8-fd6e-41c5-9de4-9c441f2ce026-kube-api-access-dp5d9\") pod \"service-ca-operator-544d749c8c-d2dsz\" (UID: \"7573b3c8-fd6e-41c5-9de4-9c441f2ce026\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" Nov 04 01:10:12.729835 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.729815 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7mxs\" (UniqueName: \"kubernetes.io/projected/a7991c77-7106-4c73-8812-ff6d4c06b666-kube-api-access-b7mxs\") pod \"volume-data-source-validator-58c9f5f46d-2w74l\" (UID: \"a7991c77-7106-4c73-8812-ff6d4c06b666\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2w74l" Nov 04 01:10:12.731236 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.731177 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl9gz\" (UniqueName: \"kubernetes.io/projected/f2df8dc5-6403-4827-b6d6-9c209c595e05-kube-api-access-jl9gz\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:12.732547 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.732526 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pbbl\" (UniqueName: \"kubernetes.io/projected/b44a2a16-b00a-4113-b0ab-5cdcb955b325-kube-api-access-2pbbl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-29mpf\" (UID: \"b44a2a16-b00a-4113-b0ab-5cdcb955b325\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" Nov 04 01:10:12.733034 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.733012 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsq8f\" (UniqueName: \"kubernetes.io/projected/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-kube-api-access-gsq8f\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:12.751021 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.750883 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" Nov 04 01:10:12.756217 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.756192 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-bound-sa-token\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.756289 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.756238 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae64edd1-43dc-406d-bcad-f7f34bb1f991-serving-cert\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:12.756289 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.756262 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-certificates\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.756583 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.756401 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e96cde4e-0238-48e6-b343-a0592f6e159e-config-volume\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:12.756583 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.756459 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.756583 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.756478 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-installation-pull-secrets\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.756583 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.756494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-bound-sa-token\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.756583 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.756521 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-trusted-ca\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.756583 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.756538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a99591bd-c354-4d6c-940d-51b0969ee916-installation-pull-secrets\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.756826 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:12.756594 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 01:10:12.756826 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:12.756610 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6556fd5cf6-2dl7j: secret "image-registry-tls" not found Nov 04 01:10:12.756826 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:12.756664 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls podName:c2aa236a-4ff4-4b39-90ff-c782721444ee nodeName:}" failed. No retries permitted until 2026-11-04 01:10:13.25664566 +0000 UTC m=+37.933073317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls") pod "image-registry-6556fd5cf6-2dl7j" (UID: "c2aa236a-4ff4-4b39-90ff-c782721444ee") : secret "image-registry-tls" not found Nov 04 01:10:12.756925 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.756842 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-certificates\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.756925 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.756878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae64edd1-43dc-406d-bcad-f7f34bb1f991-config\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:12.756925 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.756909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx7dg\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-kube-api-access-cx7dg\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.757069 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.756945 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca77e439-49cb-4ef0-9951-9564ab28ed3e-cert\") pod \"ingress-canary-ms2pt\" (UID: \"ca77e439-49cb-4ef0-9951-9564ab28ed3e\") " pod="openshift-ingress-canary/ingress-canary-ms2pt" Nov 04 01:10:12.757069 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.756982 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2aa236a-4ff4-4b39-90ff-c782721444ee-ca-trust-extracted\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.757069 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757005 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a99591bd-c354-4d6c-940d-51b0969ee916-image-registry-private-configuration\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.757069 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p78dk\" (UniqueName: \"kubernetes.io/projected/6250a398-c824-4c6e-9083-c7b0df543700-kube-api-access-p78dk\") pod \"network-check-source-6f78d59f99-dbnq4\" (UID: \"6250a398-c824-4c6e-9083-c7b0df543700\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dbnq4" Nov 04 01:10:12.757186 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757081 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-image-registry-private-configuration\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.757186 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a99591bd-c354-4d6c-940d-51b0969ee916-trusted-ca\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.757186 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757144 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-582gz\" (UID: \"5b8947a9-b2f0-4de2-a1b8-58342b6b3240\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" Nov 04 01:10:12.757186 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v22h7\" (UniqueName: \"kubernetes.io/projected/ae64edd1-43dc-406d-bcad-f7f34bb1f991-kube-api-access-v22h7\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:12.757306 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757202 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-nginx-conf\") pod \"networking-console-plugin-86c478bb88-582gz\" (UID: \"5b8947a9-b2f0-4de2-a1b8-58342b6b3240\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" Nov 04 01:10:12.757306 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757242 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a99591bd-c354-4d6c-940d-51b0969ee916-registry-certificates\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.757306 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757268 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks2wj\" (UniqueName: \"kubernetes.io/projected/e96cde4e-0238-48e6-b343-a0592f6e159e-kube-api-access-ks2wj\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:12.757392 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5dhv\" (UniqueName: \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-kube-api-access-p5dhv\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.757392 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757333 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae64edd1-43dc-406d-bcad-f7f34bb1f991-trusted-ca\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:12.757392 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757359 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:12.757505 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a99591bd-c354-4d6c-940d-51b0969ee916-ca-trust-extracted\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.757505 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk265\" (UniqueName: \"kubernetes.io/projected/ca77e439-49cb-4ef0-9951-9564ab28ed3e-kube-api-access-lk265\") pod \"ingress-canary-ms2pt\" (UID: \"ca77e439-49cb-4ef0-9951-9564ab28ed3e\") " pod="openshift-ingress-canary/ingress-canary-ms2pt" Nov 04 01:10:12.757505 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757483 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96cde4e-0238-48e6-b343-a0592f6e159e-tmp-dir\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:12.757597 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757512 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.757597 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757512 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-trusted-ca\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.757597 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757518 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae64edd1-43dc-406d-bcad-f7f34bb1f991-config\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:12.758650 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.757922 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a99591bd-c354-4d6c-940d-51b0969ee916-ca-trust-extracted\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.758650 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:12.758028 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 01:10:12.758650 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:12.758041 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-99c5d585-rzfqd: secret "image-registry-tls" not found Nov 04 01:10:12.758650 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.758089 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a99591bd-c354-4d6c-940d-51b0969ee916-registry-certificates\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.758650 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.758294 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae64edd1-43dc-406d-bcad-f7f34bb1f991-trusted-ca\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:12.758650 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:12.758368 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 01:10:12.758650 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.758386 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-nginx-conf\") pod \"networking-console-plugin-86c478bb88-582gz\" (UID: \"5b8947a9-b2f0-4de2-a1b8-58342b6b3240\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" Nov 04 01:10:12.758650 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.758398 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2aa236a-4ff4-4b39-90ff-c782721444ee-ca-trust-extracted\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.758650 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:12.758459 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert podName:5b8947a9-b2f0-4de2-a1b8-58342b6b3240 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:13.258442909 +0000 UTC m=+37.934870551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-582gz" (UID: "5b8947a9-b2f0-4de2-a1b8-58342b6b3240") : secret "networking-console-plugin-cert" not found Nov 04 01:10:12.759129 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:12.758707 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls podName:a99591bd-c354-4d6c-940d-51b0969ee916 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:13.258690097 +0000 UTC m=+37.935117740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls") pod "image-registry-99c5d585-rzfqd" (UID: "a99591bd-c354-4d6c-940d-51b0969ee916") : secret "image-registry-tls" not found Nov 04 01:10:12.759129 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.759103 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a99591bd-c354-4d6c-940d-51b0969ee916-trusted-ca\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.759744 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.759719 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a99591bd-c354-4d6c-940d-51b0969ee916-installation-pull-secrets\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.759877 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.759779 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-installation-pull-secrets\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.760308 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.760289 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae64edd1-43dc-406d-bcad-f7f34bb1f991-serving-cert\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:12.760590 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.760574 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-image-registry-private-configuration\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.761041 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.761025 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a99591bd-c354-4d6c-940d-51b0969ee916-image-registry-private-configuration\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.772436 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.771970 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2w74l" Nov 04 01:10:12.829295 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.829215 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-bound-sa-token\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.830510 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.830395 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v22h7\" (UniqueName: \"kubernetes.io/projected/ae64edd1-43dc-406d-bcad-f7f34bb1f991-kube-api-access-v22h7\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:12.830828 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.830786 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5dhv\" (UniqueName: \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-kube-api-access-p5dhv\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:12.830917 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.830874 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-bound-sa-token\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.831114 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.831094 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p78dk\" (UniqueName: \"kubernetes.io/projected/6250a398-c824-4c6e-9083-c7b0df543700-kube-api-access-p78dk\") pod \"network-check-source-6f78d59f99-dbnq4\" (UID: \"6250a398-c824-4c6e-9083-c7b0df543700\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dbnq4" Nov 04 01:10:12.832135 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.832092 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx7dg\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-kube-api-access-cx7dg\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:12.861312 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.858767 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks2wj\" (UniqueName: \"kubernetes.io/projected/e96cde4e-0238-48e6-b343-a0592f6e159e-kube-api-access-ks2wj\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:12.861312 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.858821 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:12.861312 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.858871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk265\" (UniqueName: \"kubernetes.io/projected/ca77e439-49cb-4ef0-9951-9564ab28ed3e-kube-api-access-lk265\") pod \"ingress-canary-ms2pt\" (UID: \"ca77e439-49cb-4ef0-9951-9564ab28ed3e\") " pod="openshift-ingress-canary/ingress-canary-ms2pt" Nov 04 01:10:12.861312 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.858900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96cde4e-0238-48e6-b343-a0592f6e159e-tmp-dir\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:12.861312 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:12.858954 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 01:10:12.861312 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.858964 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e96cde4e-0238-48e6-b343-a0592f6e159e-config-volume\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:12.861312 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.859066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca77e439-49cb-4ef0-9951-9564ab28ed3e-cert\") pod \"ingress-canary-ms2pt\" (UID: \"ca77e439-49cb-4ef0-9951-9564ab28ed3e\") " pod="openshift-ingress-canary/ingress-canary-ms2pt" Nov 04 01:10:12.861312 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:12.859228 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls podName:e96cde4e-0238-48e6-b343-a0592f6e159e nodeName:}" failed. No retries permitted until 2026-11-04 01:10:13.359202571 +0000 UTC m=+38.035630205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls") pod "dns-default-2h8lq" (UID: "e96cde4e-0238-48e6-b343-a0592f6e159e") : secret "dns-default-metrics-tls" not found Nov 04 01:10:12.861312 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.859329 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96cde4e-0238-48e6-b343-a0592f6e159e-tmp-dir\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:12.861312 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.859609 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e96cde4e-0238-48e6-b343-a0592f6e159e-config-volume\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:12.861880 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.861693 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca77e439-49cb-4ef0-9951-9564ab28ed3e-cert\") pod \"ingress-canary-ms2pt\" (UID: \"ca77e439-49cb-4ef0-9951-9564ab28ed3e\") " pod="openshift-ingress-canary/ingress-canary-ms2pt" Nov 04 01:10:12.907119 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.907077 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dbnq4" Nov 04 01:10:12.921880 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.921860 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-65nzk" Nov 04 01:10:12.923820 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.923800 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks2wj\" (UniqueName: \"kubernetes.io/projected/e96cde4e-0238-48e6-b343-a0592f6e159e-kube-api-access-ks2wj\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:12.923976 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.923960 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk265\" (UniqueName: \"kubernetes.io/projected/ca77e439-49cb-4ef0-9951-9564ab28ed3e-kube-api-access-lk265\") pod \"ingress-canary-ms2pt\" (UID: \"ca77e439-49cb-4ef0-9951-9564ab28ed3e\") " pod="openshift-ingress-canary/ingress-canary-ms2pt" Nov 04 01:10:12.986602 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:12.986376 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ms2pt" Nov 04 01:10:13.033331 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.033301 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" Nov 04 01:10:13.063688 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.063659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:13.063940 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:13.063921 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 01:10:13.064007 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:13.063997 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls podName:75c5b017-8f43-41dc-b79b-0353de290289 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:14.063976062 +0000 UTC m=+38.740403710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fh4kl" (UID: "75c5b017-8f43-41dc-b79b-0353de290289") : secret "cluster-monitoring-operator-tls" not found Nov 04 01:10:13.091560 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.091535 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xzhqb"] Nov 04 01:10:13.096127 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.096102 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-d26rm"] Nov 04 01:10:13.100410 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:13.100376 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b2b155_18ef_40a1_b87b_a6e44e130d65.slice/crio-5bc62a7e61ebb236bb4631d93bab8331783661fd6a433c1d3f2744be9b012835 WatchSource:0}: Error finding container 5bc62a7e61ebb236bb4631d93bab8331783661fd6a433c1d3f2744be9b012835: Status 404 returned error can't find the container with id 5bc62a7e61ebb236bb4631d93bab8331783661fd6a433c1d3f2744be9b012835 Nov 04 01:10:13.118043 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.118021 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:13.147341 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.147264 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-dgdxz"] Nov 04 01:10:13.157731 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.157636 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz"] Nov 04 01:10:13.163227 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:13.163193 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cecaa47_44ef_440e_80eb_45230fe3edc0.slice/crio-a02e524282e41a87c6ab6ad18b28a081e82927589533c29c68df9f4ea7a8a94c WatchSource:0}: Error finding container a02e524282e41a87c6ab6ad18b28a081e82927589533c29c68df9f4ea7a8a94c: Status 404 returned error can't find the container with id a02e524282e41a87c6ab6ad18b28a081e82927589533c29c68df9f4ea7a8a94c Nov 04 01:10:13.164949 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.164434 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:13.164949 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.164479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:13.164949 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:13.164547 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 01:10:13.164949 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.164593 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:13.164949 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:13.164617 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs podName:f2df8dc5-6403-4827-b6d6-9c209c595e05 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:14.164600167 +0000 UTC m=+38.841027812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs") pod "router-default-f5c95ffcf-hl6wf" (UID: "f2df8dc5-6403-4827-b6d6-9c209c595e05") : secret "router-metrics-certs-default" not found Nov 04 01:10:13.164949 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:13.164675 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 01:10:13.164949 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:13.164720 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls podName:ef7aa61c-2789-49ed-989d-fa77fed0ff0f nodeName:}" failed. No retries permitted until 2026-11-04 01:10:14.164705064 +0000 UTC m=+38.841132711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q449k" (UID: "ef7aa61c-2789-49ed-989d-fa77fed0ff0f") : secret "samples-operator-tls" not found Nov 04 01:10:13.164949 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:13.164820 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle podName:f2df8dc5-6403-4827-b6d6-9c209c595e05 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:14.164809912 +0000 UTC m=+38.841237559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle") pod "router-default-f5c95ffcf-hl6wf" (UID: "f2df8dc5-6403-4827-b6d6-9c209c595e05") : configmap references non-existent config key: service-ca.crt Nov 04 01:10:13.170797 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:13.170775 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7573b3c8_fd6e_41c5_9de4_9c441f2ce026.slice/crio-86a37a601387a008f2c0ddd8f57e96ab06eb284bd0203263892fec8bfbd4aec4 WatchSource:0}: Error finding container 86a37a601387a008f2c0ddd8f57e96ab06eb284bd0203263892fec8bfbd4aec4: Status 404 returned error can't find the container with id 86a37a601387a008f2c0ddd8f57e96ab06eb284bd0203263892fec8bfbd4aec4 Nov 04 01:10:13.195271 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.195157 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2w74l"] Nov 04 01:10:13.202243 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:13.202214 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7991c77_7106_4c73_8812_ff6d4c06b666.slice/crio-f2dd755b06f3e13f4f7881c0031f294ae2d40a05f702199edab0368f01ee59c6 WatchSource:0}: Error finding container f2dd755b06f3e13f4f7881c0031f294ae2d40a05f702199edab0368f01ee59c6: Status 404 returned error can't find the container with id f2dd755b06f3e13f4f7881c0031f294ae2d40a05f702199edab0368f01ee59c6 Nov 04 01:10:13.265410 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.265384 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-582gz\" (UID: \"5b8947a9-b2f0-4de2-a1b8-58342b6b3240\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" Nov 04 01:10:13.265550 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:13.265530 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 01:10:13.265621 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.265573 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:13.265621 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:13.265599 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert podName:5b8947a9-b2f0-4de2-a1b8-58342b6b3240 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:14.265580835 +0000 UTC m=+38.942008480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-582gz" (UID: "5b8947a9-b2f0-4de2-a1b8-58342b6b3240") : secret "networking-console-plugin-cert" not found Nov 04 01:10:13.265736 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:13.265645 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 01:10:13.265736 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:13.265657 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-99c5d585-rzfqd: secret "image-registry-tls" not found Nov 04 01:10:13.265736 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.265680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:13.265736 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:13.265695 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls podName:a99591bd-c354-4d6c-940d-51b0969ee916 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:14.26568505 +0000 UTC m=+38.942112689 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls") pod "image-registry-99c5d585-rzfqd" (UID: "a99591bd-c354-4d6c-940d-51b0969ee916") : secret "image-registry-tls" not found Nov 04 01:10:13.265736 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:13.265748 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 01:10:13.265736 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:13.265762 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6556fd5cf6-2dl7j: secret "image-registry-tls" not found Nov 04 01:10:13.265964 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:13.265803 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls podName:c2aa236a-4ff4-4b39-90ff-c782721444ee nodeName:}" failed. No retries permitted until 2026-11-04 01:10:14.265791049 +0000 UTC m=+38.942218690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls") pod "image-registry-6556fd5cf6-2dl7j" (UID: "c2aa236a-4ff4-4b39-90ff-c782721444ee") : secret "image-registry-tls" not found Nov 04 01:10:13.300834 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.300806 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-dbnq4"] Nov 04 01:10:13.307807 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:13.307782 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6250a398_c824_4c6e_9083_c7b0df543700.slice/crio-4a82c2a760402ddb4b1ed8bacf6330c70ac99acd53cf7e314cc7d25d2280272e WatchSource:0}: Error finding container 4a82c2a760402ddb4b1ed8bacf6330c70ac99acd53cf7e314cc7d25d2280272e: Status 404 returned error can't find the container with id 4a82c2a760402ddb4b1ed8bacf6330c70ac99acd53cf7e314cc7d25d2280272e Nov 04 01:10:13.323347 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.323323 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-65nzk"] Nov 04 01:10:13.330439 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:13.330405 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23c835a_aacb_4f92_a2cd_799f5e870b85.slice/crio-a8df2f8562b1d8be2e8110165044fe414fe196d1fa649d8d0218c9d4dce16a11 WatchSource:0}: Error finding container a8df2f8562b1d8be2e8110165044fe414fe196d1fa649d8d0218c9d4dce16a11: Status 404 returned error can't find the container with id a8df2f8562b1d8be2e8110165044fe414fe196d1fa649d8d0218c9d4dce16a11 Nov 04 01:10:13.366554 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.366529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:13.366848 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:13.366828 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 01:10:13.366952 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:13.366891 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls podName:e96cde4e-0238-48e6-b343-a0592f6e159e nodeName:}" failed. No retries permitted until 2026-11-04 01:10:14.366873852 +0000 UTC m=+39.043301496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls") pod "dns-default-2h8lq" (UID: "e96cde4e-0238-48e6-b343-a0592f6e159e") : secret "dns-default-metrics-tls" not found Nov 04 01:10:13.372867 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.372846 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ms2pt"] Nov 04 01:10:13.379610 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:13.379584 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca77e439_49cb_4ef0_9951_9564ab28ed3e.slice/crio-251e21ccd12219fcf62ea2f806f78dc2c3356ebc03f66d402fbcd980dd360e8a WatchSource:0}: Error finding container 251e21ccd12219fcf62ea2f806f78dc2c3356ebc03f66d402fbcd980dd360e8a: Status 404 returned error can't find the container with id 251e21ccd12219fcf62ea2f806f78dc2c3356ebc03f66d402fbcd980dd360e8a Nov 04 01:10:13.425661 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.425639 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf"] Nov 04 01:10:13.433789 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:13.433762 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44a2a16_b00a_4113_b0ab_5cdcb955b325.slice/crio-cfb374093c7ff29f6badcdd2465ae551ee208b14450e298c52bc5a9384416355 WatchSource:0}: Error finding container cfb374093c7ff29f6badcdd2465ae551ee208b14450e298c52bc5a9384416355: Status 404 returned error can't find the container with id cfb374093c7ff29f6badcdd2465ae551ee208b14450e298c52bc5a9384416355 Nov 04 01:10:13.536096 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.536068 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-8cfg7"] Nov 04 01:10:13.543134 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:13.543114 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae64edd1_43dc_406d_bcad_f7f34bb1f991.slice/crio-36d411089d38ab9155869fc5fc027a051a82de1059856529daf993a1c77e0a67 WatchSource:0}: Error finding container 36d411089d38ab9155869fc5fc027a051a82de1059856529daf993a1c77e0a67: Status 404 returned error can't find the container with id 36d411089d38ab9155869fc5fc027a051a82de1059856529daf993a1c77e0a67 Nov 04 01:10:13.820470 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.820446 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:10:13.820611 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.820450 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:10:13.820688 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.820640 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:13.873699 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.873677 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 01:10:13.874061 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.873759 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-8hhs6\"" Nov 04 01:10:13.874061 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.873783 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8lz6w\"" Nov 04 01:10:13.874061 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.873810 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-x4nsn\"" Nov 04 01:10:13.898033 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:13.898012 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 01:10:14.001468 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.001441 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dbnq4" event={"ID":"6250a398-c824-4c6e-9083-c7b0df543700","Type":"ContainerStarted","Data":"4a82c2a760402ddb4b1ed8bacf6330c70ac99acd53cf7e314cc7d25d2280272e"} Nov 04 01:10:14.002518 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.002494 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-dgdxz" event={"ID":"5cecaa47-44ef-440e-80eb-45230fe3edc0","Type":"ContainerStarted","Data":"a02e524282e41a87c6ab6ad18b28a081e82927589533c29c68df9f4ea7a8a94c"} Nov 04 01:10:14.003602 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.003582 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ms2pt" event={"ID":"ca77e439-49cb-4ef0-9951-9564ab28ed3e","Type":"ContainerStarted","Data":"251e21ccd12219fcf62ea2f806f78dc2c3356ebc03f66d402fbcd980dd360e8a"} Nov 04 01:10:14.004614 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.004587 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" event={"ID":"7573b3c8-fd6e-41c5-9de4-9c441f2ce026","Type":"ContainerStarted","Data":"86a37a601387a008f2c0ddd8f57e96ab06eb284bd0203263892fec8bfbd4aec4"} Nov 04 01:10:14.005901 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.005878 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-d26rm" event={"ID":"cd605a92-f88c-40ea-bde7-53c2da66a576","Type":"ContainerStarted","Data":"b3f9c95ae9b1eb5e940bad7eb3a362343db650d6a77aca25e64f885765158caa"} Nov 04 01:10:14.006025 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.005921 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-d26rm" event={"ID":"cd605a92-f88c-40ea-bde7-53c2da66a576","Type":"ContainerStarted","Data":"6c93d8fe3521e9d82ee6651bc3b234e56a2e47a042a2b191d87d595990d2def2"} Nov 04 01:10:14.006997 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.006976 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" event={"ID":"ae64edd1-43dc-406d-bcad-f7f34bb1f991","Type":"ContainerStarted","Data":"36d411089d38ab9155869fc5fc027a051a82de1059856529daf993a1c77e0a67"} Nov 04 01:10:14.009918 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.009890 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" event={"ID":"b44a2a16-b00a-4113-b0ab-5cdcb955b325","Type":"ContainerStarted","Data":"cfb374093c7ff29f6badcdd2465ae551ee208b14450e298c52bc5a9384416355"} Nov 04 01:10:14.011261 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.011242 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2w74l" event={"ID":"a7991c77-7106-4c73-8812-ff6d4c06b666","Type":"ContainerStarted","Data":"f2dd755b06f3e13f4f7881c0031f294ae2d40a05f702199edab0368f01ee59c6"} Nov 04 01:10:14.012681 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.012649 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xzhqb" event={"ID":"a8b2b155-18ef-40a1-b87b-a6e44e130d65","Type":"ContainerStarted","Data":"c5193b82dd984357ca504d6bbf82d99eef4875aba09f766cc6153d0ccc9dd697"} Nov 04 01:10:14.012775 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.012689 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xzhqb" event={"ID":"a8b2b155-18ef-40a1-b87b-a6e44e130d65","Type":"ContainerStarted","Data":"5bc62a7e61ebb236bb4631d93bab8331783661fd6a433c1d3f2744be9b012835"} Nov 04 01:10:14.013934 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.013915 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-65nzk" event={"ID":"a23c835a-aacb-4f92-a2cd-799f5e870b85","Type":"ContainerStarted","Data":"f0e03ce1075589788174429fa9b414dadd9804960bd2f9f1804fc02a25a38908"} Nov 04 01:10:14.013991 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.013950 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-65nzk" event={"ID":"a23c835a-aacb-4f92-a2cd-799f5e870b85","Type":"ContainerStarted","Data":"a8df2f8562b1d8be2e8110165044fe414fe196d1fa649d8d0218c9d4dce16a11"} Nov 04 01:10:14.074456 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.074394 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:14.074556 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:14.074544 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 01:10:14.074597 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:14.074592 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls podName:75c5b017-8f43-41dc-b79b-0353de290289 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:16.074579713 +0000 UTC m=+40.751007352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fh4kl" (UID: "75c5b017-8f43-41dc-b79b-0353de290289") : secret "cluster-monitoring-operator-tls" not found Nov 04 01:10:14.175418 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.175386 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:14.175550 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.175526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:14.175550 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:14.175538 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 01:10:14.175659 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:14.175588 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls podName:ef7aa61c-2789-49ed-989d-fa77fed0ff0f nodeName:}" failed. No retries permitted until 2026-11-04 01:10:16.175574913 +0000 UTC m=+40.852002554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q449k" (UID: "ef7aa61c-2789-49ed-989d-fa77fed0ff0f") : secret "samples-operator-tls" not found Nov 04 01:10:14.175659 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:14.175625 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 01:10:14.175764 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:14.175682 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs podName:f2df8dc5-6403-4827-b6d6-9c209c595e05 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:16.175666079 +0000 UTC m=+40.852093725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs") pod "router-default-f5c95ffcf-hl6wf" (UID: "f2df8dc5-6403-4827-b6d6-9c209c595e05") : secret "router-metrics-certs-default" not found Nov 04 01:10:14.175764 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.175706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:14.175873 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:14.175860 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle podName:f2df8dc5-6403-4827-b6d6-9c209c595e05 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:16.175848519 +0000 UTC m=+40.852276148 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle") pod "router-default-f5c95ffcf-hl6wf" (UID: "f2df8dc5-6403-4827-b6d6-9c209c595e05") : configmap references non-existent config key: service-ca.crt Nov 04 01:10:14.276178 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.276158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-582gz\" (UID: \"5b8947a9-b2f0-4de2-a1b8-58342b6b3240\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" Nov 04 01:10:14.276262 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.276231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:14.276311 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:14.276302 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 01:10:14.276359 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:14.276313 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-99c5d585-rzfqd: secret "image-registry-tls" not found Nov 04 01:10:14.276359 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.276330 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:14.276359 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:14.276345 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls podName:a99591bd-c354-4d6c-940d-51b0969ee916 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:16.27633504 +0000 UTC m=+40.952762685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls") pod "image-registry-99c5d585-rzfqd" (UID: "a99591bd-c354-4d6c-940d-51b0969ee916") : secret "image-registry-tls" not found Nov 04 01:10:14.276548 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:14.276300 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 01:10:14.276548 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:14.276444 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 01:10:14.276548 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:14.276455 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert podName:5b8947a9-b2f0-4de2-a1b8-58342b6b3240 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:16.276439568 +0000 UTC m=+40.952867209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-582gz" (UID: "5b8947a9-b2f0-4de2-a1b8-58342b6b3240") : secret "networking-console-plugin-cert" not found Nov 04 01:10:14.276548 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:14.276458 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6556fd5cf6-2dl7j: secret "image-registry-tls" not found Nov 04 01:10:14.276548 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:14.276495 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls podName:c2aa236a-4ff4-4b39-90ff-c782721444ee nodeName:}" failed. No retries permitted until 2026-11-04 01:10:16.276484215 +0000 UTC m=+40.952911857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls") pod "image-registry-6556fd5cf6-2dl7j" (UID: "c2aa236a-4ff4-4b39-90ff-c782721444ee") : secret "image-registry-tls" not found Nov 04 01:10:14.376881 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.376820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:14.376975 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:14.376957 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 01:10:14.377011 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:14.376993 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls podName:e96cde4e-0238-48e6-b343-a0592f6e159e nodeName:}" failed. No retries permitted until 2026-11-04 01:10:16.376983459 +0000 UTC m=+41.053411084 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls") pod "dns-default-2h8lq" (UID: "e96cde4e-0238-48e6-b343-a0592f6e159e") : secret "dns-default-metrics-tls" not found Nov 04 01:10:14.492112 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.492046 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-65nzk" podStartSLOduration=451.492029155 podStartE2EDuration="7m31.492029155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:14.49192086 +0000 UTC m=+39.168348508" watchObservedRunningTime="2026-11-04 01:10:14.492029155 +0000 UTC m=+39.168456806" Nov 04 01:10:14.608594 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:14.608542 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xzhqb" podStartSLOduration=451.608504644 podStartE2EDuration="7m31.608504644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:14.607304249 +0000 UTC m=+39.283731896" watchObservedRunningTime="2026-11-04 01:10:14.608504644 +0000 UTC m=+39.284932294" Nov 04 01:10:16.022611 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:16.022379 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2w74l" event={"ID":"a7991c77-7106-4c73-8812-ff6d4c06b666","Type":"ContainerStarted","Data":"cbd210da5f2d97feed5a7b49d2767a2b7056178641bf14fa637437974b6de10d"} Nov 04 01:10:16.095644 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:16.095611 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:16.095937 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:16.095921 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 01:10:16.096017 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:16.095982 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls podName:75c5b017-8f43-41dc-b79b-0353de290289 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:20.095964698 +0000 UTC m=+44.772392327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fh4kl" (UID: "75c5b017-8f43-41dc-b79b-0353de290289") : secret "cluster-monitoring-operator-tls" not found Nov 04 01:10:16.197281 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:16.197243 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:16.197455 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:16.197308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:16.197455 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:16.197404 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 01:10:16.197455 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:16.197445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:16.197586 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:16.197483 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle podName:f2df8dc5-6403-4827-b6d6-9c209c595e05 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:20.197464726 +0000 UTC m=+44.873892365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle") pod "router-default-f5c95ffcf-hl6wf" (UID: "f2df8dc5-6403-4827-b6d6-9c209c595e05") : configmap references non-existent config key: service-ca.crt Nov 04 01:10:16.197586 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:16.197520 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs podName:f2df8dc5-6403-4827-b6d6-9c209c595e05 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:20.197510352 +0000 UTC m=+44.873937982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs") pod "router-default-f5c95ffcf-hl6wf" (UID: "f2df8dc5-6403-4827-b6d6-9c209c595e05") : secret "router-metrics-certs-default" not found Nov 04 01:10:16.197586 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:16.197545 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 01:10:16.197586 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:16.197587 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls podName:ef7aa61c-2789-49ed-989d-fa77fed0ff0f nodeName:}" failed. No retries permitted until 2026-11-04 01:10:20.197575283 +0000 UTC m=+44.874002909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q449k" (UID: "ef7aa61c-2789-49ed-989d-fa77fed0ff0f") : secret "samples-operator-tls" not found Nov 04 01:10:16.298686 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:16.298647 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-582gz\" (UID: \"5b8947a9-b2f0-4de2-a1b8-58342b6b3240\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" Nov 04 01:10:16.298838 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:16.298751 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:16.298838 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:16.298803 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 01:10:16.298940 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:16.298877 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert podName:5b8947a9-b2f0-4de2-a1b8-58342b6b3240 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:20.298855883 +0000 UTC m=+44.975283524 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-582gz" (UID: "5b8947a9-b2f0-4de2-a1b8-58342b6b3240") : secret "networking-console-plugin-cert" not found Nov 04 01:10:16.298940 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:16.298809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:16.298940 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:16.298901 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 01:10:16.298940 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:16.298915 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6556fd5cf6-2dl7j: secret "image-registry-tls" not found Nov 04 01:10:16.299086 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:16.298949 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls podName:c2aa236a-4ff4-4b39-90ff-c782721444ee nodeName:}" failed. No retries permitted until 2026-11-04 01:10:20.298939978 +0000 UTC m=+44.975367603 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls") pod "image-registry-6556fd5cf6-2dl7j" (UID: "c2aa236a-4ff4-4b39-90ff-c782721444ee") : secret "image-registry-tls" not found Nov 04 01:10:16.299086 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:16.299008 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 01:10:16.299086 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:16.299021 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-99c5d585-rzfqd: secret "image-registry-tls" not found Nov 04 01:10:16.299086 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:16.299070 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls podName:a99591bd-c354-4d6c-940d-51b0969ee916 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:20.299056289 +0000 UTC m=+44.975483943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls") pod "image-registry-99c5d585-rzfqd" (UID: "a99591bd-c354-4d6c-940d-51b0969ee916") : secret "image-registry-tls" not found Nov 04 01:10:16.399947 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:16.399911 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:16.400143 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:16.400098 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 01:10:16.400220 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:16.400156 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls podName:e96cde4e-0238-48e6-b343-a0592f6e159e nodeName:}" failed. No retries permitted until 2026-11-04 01:10:20.400140451 +0000 UTC m=+45.076568094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls") pod "dns-default-2h8lq" (UID: "e96cde4e-0238-48e6-b343-a0592f6e159e") : secret "dns-default-metrics-tls" not found Nov 04 01:10:17.146425 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:17.146341 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2w74l" podStartSLOduration=423.500123623 podStartE2EDuration="7m6.146321484s" totalImagesPullingTime="2.646197861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:17.14533228 +0000 UTC m=+41.821759928" watchObservedRunningTime="2026-11-04 01:10:17.146321484 +0000 UTC m=+41.822749136" Nov 04 01:10:17.147136 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:17.147004 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-d26rm" podStartSLOduration=454.14699155 podStartE2EDuration="7m34.14699155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:14.72492238 +0000 UTC m=+39.401350029" watchObservedRunningTime="2026-11-04 01:10:17.14699155 +0000 UTC m=+41.823419210" Nov 04 01:10:18.034488 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:18.034453 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-dgdxz" event={"ID":"5cecaa47-44ef-440e-80eb-45230fe3edc0","Type":"ContainerStarted","Data":"5f8aeb58ea3d2c92298c966553bf407cf70f4b4754e2a66bf668ade4ec92ed53"} Nov 04 01:10:18.157504 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:18.156690 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-dgdxz" podStartSLOduration=422.450038307 podStartE2EDuration="7m7.156671857s" totalImagesPullingTime="4.70663355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:18.156005759 +0000 UTC m=+42.832433405" watchObservedRunningTime="2026-11-04 01:10:18.156671857 +0000 UTC m=+42.833099505" Nov 04 01:10:19.039101 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.038935 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/0.log" Nov 04 01:10:19.039255 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.039108 2580 generic.go:356] "Generic (PLEG): container finished" podID="ae64edd1-43dc-406d-bcad-f7f34bb1f991" containerID="326bdb69045627a5f11f24bcc0e02104ba997663a541739c239f19b0ad35fdb3" exitCode=255 Nov 04 01:10:19.039255 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.039184 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" event={"ID":"ae64edd1-43dc-406d-bcad-f7f34bb1f991","Type":"ContainerDied","Data":"326bdb69045627a5f11f24bcc0e02104ba997663a541739c239f19b0ad35fdb3"} Nov 04 01:10:19.039362 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.039350 2580 scope.go:122] "RemoveContainer" containerID="326bdb69045627a5f11f24bcc0e02104ba997663a541739c239f19b0ad35fdb3" Nov 04 01:10:19.265675 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.265644 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4j876"] Nov 04 01:10:19.289974 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.289911 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:19.328981 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.326360 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:19.328981 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.326538 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:19.328981 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.326596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tgzg\" (UniqueName: \"kubernetes.io/projected/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-api-access-2tgzg\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:19.328981 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.326640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5dd40be-e515-4fe8-be55-b93f5b97973e-data-volume\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:19.328981 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.326735 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5dd40be-e515-4fe8-be55-b93f5b97973e-crio-socket\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:19.332218 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.332193 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4j876"] Nov 04 01:10:19.343662 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.343641 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 01:10:19.343837 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.343815 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 01:10:19.343925 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.343872 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qr45v\"" Nov 04 01:10:19.427462 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.427392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:19.427603 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:19.427509 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 01:10:19.427603 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.427563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:19.427603 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:19.427584 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm podName:c5dd40be-e515-4fe8-be55-b93f5b97973e nodeName:}" failed. No retries permitted until 2026-11-04 01:10:19.927564881 +0000 UTC m=+44.603992511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4j876" (UID: "c5dd40be-e515-4fe8-be55-b93f5b97973e") : configmap "kube-rbac-proxy" not found Nov 04 01:10:19.427767 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.427635 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tgzg\" (UniqueName: \"kubernetes.io/projected/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-api-access-2tgzg\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:19.427767 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:19.427658 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 01:10:19.427767 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.427677 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5dd40be-e515-4fe8-be55-b93f5b97973e-data-volume\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:19.427767 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:19.427708 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls podName:c5dd40be-e515-4fe8-be55-b93f5b97973e nodeName:}" failed. No retries permitted until 2026-11-04 01:10:19.927693039 +0000 UTC m=+44.604120671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4j876" (UID: "c5dd40be-e515-4fe8-be55-b93f5b97973e") : secret "insights-runtime-extractor-tls" not found Nov 04 01:10:19.427950 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.427793 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5dd40be-e515-4fe8-be55-b93f5b97973e-crio-socket\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:19.427988 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.427960 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5dd40be-e515-4fe8-be55-b93f5b97973e-data-volume\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:19.427988 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.427975 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5dd40be-e515-4fe8-be55-b93f5b97973e-crio-socket\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:19.538301 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.538281 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-jmkw9-6xlv6"] Nov 04 01:10:19.546343 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.546300 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tgzg\" (UniqueName: \"kubernetes.io/projected/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-api-access-2tgzg\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:19.567602 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.567585 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:19.597006 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.596985 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jmkw9-6xlv6"] Nov 04 01:10:19.629187 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.629164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-827hl\" (UniqueName: \"kubernetes.io/projected/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-kube-api-access-827hl\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:19.629301 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.629219 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-trusted-ca-bundle\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:19.629360 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.629345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-service-ca-bundle\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:19.629393 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.629375 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-archives-path\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:19.729984 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.729959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-archives-path\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:19.730113 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.730072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-827hl\" (UniqueName: \"kubernetes.io/projected/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-kube-api-access-827hl\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:19.730173 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.730154 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-trusted-ca-bundle\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:19.730287 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.730269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-service-ca-bundle\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:19.730479 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.730459 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-archives-path\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:19.730816 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.730801 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-service-ca-bundle\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:19.731063 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.731041 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-trusted-ca-bundle\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:19.792125 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.792104 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-827hl\" (UniqueName: \"kubernetes.io/projected/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-kube-api-access-827hl\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:19.891261 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.891198 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:19.932480 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.932457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:19.932609 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:19.932593 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 01:10:19.932672 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:19.932616 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:19.932672 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:19.932657 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls podName:c5dd40be-e515-4fe8-be55-b93f5b97973e nodeName:}" failed. No retries permitted until 2026-11-04 01:10:20.932638104 +0000 UTC m=+45.609065746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4j876" (UID: "c5dd40be-e515-4fe8-be55-b93f5b97973e") : secret "insights-runtime-extractor-tls" not found Nov 04 01:10:19.932765 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:19.932683 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 01:10:19.932765 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:19.932736 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm podName:c5dd40be-e515-4fe8-be55-b93f5b97973e nodeName:}" failed. No retries permitted until 2026-11-04 01:10:20.932722408 +0000 UTC m=+45.609150057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4j876" (UID: "c5dd40be-e515-4fe8-be55-b93f5b97973e") : configmap "kube-rbac-proxy" not found Nov 04 01:10:20.043733 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.043501 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dbnq4" event={"ID":"6250a398-c824-4c6e-9083-c7b0df543700","Type":"ContainerStarted","Data":"eacdc3d1d28e1a02a340783738d3adfc3eb052648099192278c25571b55d5669"} Nov 04 01:10:20.045079 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.045055 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ms2pt" event={"ID":"ca77e439-49cb-4ef0-9951-9564ab28ed3e","Type":"ContainerStarted","Data":"031f5f182a9322f8b274c3fcb10893851c2fdfdfd4eadfb762ee7eee69e733d3"} Nov 04 01:10:20.046912 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.046557 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" event={"ID":"7573b3c8-fd6e-41c5-9de4-9c441f2ce026","Type":"ContainerStarted","Data":"400a166532130453e6a7e06da907398203e1d01fbbb7a5154b87d9bd35525ed1"} Nov 04 01:10:20.048440 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.048406 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:10:20.048840 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.048823 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/0.log" Nov 04 01:10:20.048935 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.048852 2580 generic.go:356] "Generic (PLEG): container finished" podID="ae64edd1-43dc-406d-bcad-f7f34bb1f991" containerID="86f7eda2eef130111d4393a379809aa2517d0821e73d020723ac0d438c164361" exitCode=255 Nov 04 01:10:20.048935 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.048910 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" event={"ID":"ae64edd1-43dc-406d-bcad-f7f34bb1f991","Type":"ContainerDied","Data":"86f7eda2eef130111d4393a379809aa2517d0821e73d020723ac0d438c164361"} Nov 04 01:10:20.049047 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.048938 2580 scope.go:122] "RemoveContainer" containerID="326bdb69045627a5f11f24bcc0e02104ba997663a541739c239f19b0ad35fdb3" Nov 04 01:10:20.049345 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.049119 2580 scope.go:122] "RemoveContainer" containerID="86f7eda2eef130111d4393a379809aa2517d0821e73d020723ac0d438c164361" Nov 04 01:10:20.049345 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.049266 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-8cfg7_openshift-console-operator(ae64edd1-43dc-406d-bcad-f7f34bb1f991)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" podUID="ae64edd1-43dc-406d-bcad-f7f34bb1f991" Nov 04 01:10:20.050583 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.050312 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" event={"ID":"b44a2a16-b00a-4113-b0ab-5cdcb955b325","Type":"ContainerStarted","Data":"e4691df3ebc0757feac0cac105d84bd2f1b27c2a7372a7d0a3db2ba79314143e"} Nov 04 01:10:20.133835 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.133806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:20.134017 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.133999 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 01:10:20.134098 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.134074 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls podName:75c5b017-8f43-41dc-b79b-0353de290289 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:28.134057028 +0000 UTC m=+52.810484685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fh4kl" (UID: "75c5b017-8f43-41dc-b79b-0353de290289") : secret "cluster-monitoring-operator-tls" not found Nov 04 01:10:20.159942 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.159875 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dbnq4" podStartSLOduration=421.943702778 podStartE2EDuration="7m8.159863629s" totalImagesPullingTime="6.216160851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:20.15948885 +0000 UTC m=+44.835916493" watchObservedRunningTime="2026-11-04 01:10:20.159863629 +0000 UTC m=+44.836291276" Nov 04 01:10:20.235077 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.235054 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:20.235176 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.235087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:20.235176 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.235163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:20.235240 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.235187 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 01:10:20.235275 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.235243 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 01:10:20.235275 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.235266 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs podName:f2df8dc5-6403-4827-b6d6-9c209c595e05 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:28.235249005 +0000 UTC m=+52.911676647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs") pod "router-default-f5c95ffcf-hl6wf" (UID: "f2df8dc5-6403-4827-b6d6-9c209c595e05") : secret "router-metrics-certs-default" not found Nov 04 01:10:20.235348 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.235280 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls podName:ef7aa61c-2789-49ed-989d-fa77fed0ff0f nodeName:}" failed. No retries permitted until 2026-11-04 01:10:28.235274546 +0000 UTC m=+52.911702172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q449k" (UID: "ef7aa61c-2789-49ed-989d-fa77fed0ff0f") : secret "samples-operator-tls" not found Nov 04 01:10:20.235348 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.235292 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle podName:f2df8dc5-6403-4827-b6d6-9c209c595e05 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:28.235286953 +0000 UTC m=+52.911714622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle") pod "router-default-f5c95ffcf-hl6wf" (UID: "f2df8dc5-6403-4827-b6d6-9c209c595e05") : configmap references non-existent config key: service-ca.crt Nov 04 01:10:20.280764 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.280737 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jmkw9-6xlv6"] Nov 04 01:10:20.288238 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:20.288212 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f6e24e_f98d_4b9a_8d51_080c2c38cf29.slice/crio-83ffd8448ded9b9ec3ef6209f1bf5d4b96993fb173ab544e4321f7611f36e6ce WatchSource:0}: Error finding container 83ffd8448ded9b9ec3ef6209f1bf5d4b96993fb173ab544e4321f7611f36e6ce: Status 404 returned error can't find the container with id 83ffd8448ded9b9ec3ef6209f1bf5d4b96993fb173ab544e4321f7611f36e6ce Nov 04 01:10:20.327713 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.327441 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" podStartSLOduration=423.371098217 podStartE2EDuration="7m9.327409748s" totalImagesPullingTime="5.956311531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:20.326908537 +0000 UTC m=+45.003336184" watchObservedRunningTime="2026-11-04 01:10:20.327409748 +0000 UTC m=+45.003837397" Nov 04 01:10:20.335729 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.335710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:20.335841 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.335768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:20.335898 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.335858 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 01:10:20.335898 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.335876 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-99c5d585-rzfqd: secret "image-registry-tls" not found Nov 04 01:10:20.336002 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.335891 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-582gz\" (UID: \"5b8947a9-b2f0-4de2-a1b8-58342b6b3240\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" Nov 04 01:10:20.336002 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.335937 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls podName:a99591bd-c354-4d6c-940d-51b0969ee916 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:28.335920388 +0000 UTC m=+53.012348028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls") pod "image-registry-99c5d585-rzfqd" (UID: "a99591bd-c354-4d6c-940d-51b0969ee916") : secret "image-registry-tls" not found Nov 04 01:10:20.336002 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.335943 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 01:10:20.336002 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.335957 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6556fd5cf6-2dl7j: secret "image-registry-tls" not found Nov 04 01:10:20.336002 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.335992 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 01:10:20.336216 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.336010 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls podName:c2aa236a-4ff4-4b39-90ff-c782721444ee nodeName:}" failed. No retries permitted until 2026-11-04 01:10:28.335999021 +0000 UTC m=+53.012426658 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls") pod "image-registry-6556fd5cf6-2dl7j" (UID: "c2aa236a-4ff4-4b39-90ff-c782721444ee") : secret "image-registry-tls" not found Nov 04 01:10:20.336216 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.336042 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert podName:5b8947a9-b2f0-4de2-a1b8-58342b6b3240 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:28.336024019 +0000 UTC m=+53.012451664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-582gz" (UID: "5b8947a9-b2f0-4de2-a1b8-58342b6b3240") : secret "networking-console-plugin-cert" not found Nov 04 01:10:20.437242 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.437153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:20.437368 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.437301 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 01:10:20.437368 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.437365 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls podName:e96cde4e-0238-48e6-b343-a0592f6e159e nodeName:}" failed. No retries permitted until 2026-11-04 01:10:28.437351095 +0000 UTC m=+53.113778720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls") pod "dns-default-2h8lq" (UID: "e96cde4e-0238-48e6-b343-a0592f6e159e") : secret "dns-default-metrics-tls" not found Nov 04 01:10:20.488854 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.488806 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" podStartSLOduration=423.795120394 podStartE2EDuration="7m9.488791335s" totalImagesPullingTime="5.693670941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:20.48792303 +0000 UTC m=+45.164350678" watchObservedRunningTime="2026-11-04 01:10:20.488791335 +0000 UTC m=+45.165218981" Nov 04 01:10:20.610026 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.609975 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ms2pt" podStartSLOduration=2.847303583 podStartE2EDuration="8.609965671s" totalImagesPullingTime="5.762662088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:20.609281956 +0000 UTC m=+45.285709603" watchObservedRunningTime="2026-11-04 01:10:20.609965671 +0000 UTC m=+45.286393317" Nov 04 01:10:20.760430 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.760394 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vkk6b"] Nov 04 01:10:20.766881 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.766862 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vkk6b" Nov 04 01:10:20.772479 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.772459 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-db6x7"] Nov 04 01:10:20.779724 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.779702 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db6x7" Nov 04 01:10:20.817820 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.817795 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vkk6b"] Nov 04 01:10:20.824808 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.824789 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 04 01:10:20.824808 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.824789 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ggznq\"" Nov 04 01:10:20.824966 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.824952 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 04 01:10:20.830944 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.830922 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-db6x7"] Nov 04 01:10:20.841246 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.841226 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b5fm\" (UniqueName: \"kubernetes.io/projected/c60eb5db-ec53-40bb-9a72-c712cec7c32c-kube-api-access-4b5fm\") pod \"migrator-66df5d496b-db6x7\" (UID: \"c60eb5db-ec53-40bb-9a72-c712cec7c32c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db6x7" Nov 04 01:10:20.841383 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.841366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxsvl\" (UniqueName: \"kubernetes.io/projected/8ddedc2c-735e-41e5-a5a9-3386ed890ee1-kube-api-access-kxsvl\") pod \"migrator-66df5d496b-vkk6b\" (UID: \"8ddedc2c-735e-41e5-a5a9-3386ed890ee1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vkk6b" Nov 04 01:10:20.942570 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.942542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:20.942700 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.942590 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxsvl\" (UniqueName: \"kubernetes.io/projected/8ddedc2c-735e-41e5-a5a9-3386ed890ee1-kube-api-access-kxsvl\") pod \"migrator-66df5d496b-vkk6b\" (UID: \"8ddedc2c-735e-41e5-a5a9-3386ed890ee1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vkk6b" Nov 04 01:10:20.942700 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.942659 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 01:10:20.942700 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.942692 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:20.942872 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.942723 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm podName:c5dd40be-e515-4fe8-be55-b93f5b97973e nodeName:}" failed. No retries permitted until 2026-11-04 01:10:22.942702858 +0000 UTC m=+47.619130502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4j876" (UID: "c5dd40be-e515-4fe8-be55-b93f5b97973e") : configmap "kube-rbac-proxy" not found Nov 04 01:10:20.942872 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.942779 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 01:10:20.942872 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:20.942793 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b5fm\" (UniqueName: \"kubernetes.io/projected/c60eb5db-ec53-40bb-9a72-c712cec7c32c-kube-api-access-4b5fm\") pod \"migrator-66df5d496b-db6x7\" (UID: \"c60eb5db-ec53-40bb-9a72-c712cec7c32c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db6x7" Nov 04 01:10:20.942872 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:20.942853 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls podName:c5dd40be-e515-4fe8-be55-b93f5b97973e nodeName:}" failed. No retries permitted until 2026-11-04 01:10:22.942823932 +0000 UTC m=+47.619251578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4j876" (UID: "c5dd40be-e515-4fe8-be55-b93f5b97973e") : secret "insights-runtime-extractor-tls" not found Nov 04 01:10:21.008395 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.008364 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxsvl\" (UniqueName: \"kubernetes.io/projected/8ddedc2c-735e-41e5-a5a9-3386ed890ee1-kube-api-access-kxsvl\") pod \"migrator-66df5d496b-vkk6b\" (UID: \"8ddedc2c-735e-41e5-a5a9-3386ed890ee1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vkk6b" Nov 04 01:10:21.043245 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.043190 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b5fm\" (UniqueName: \"kubernetes.io/projected/c60eb5db-ec53-40bb-9a72-c712cec7c32c-kube-api-access-4b5fm\") pod \"migrator-66df5d496b-db6x7\" (UID: \"c60eb5db-ec53-40bb-9a72-c712cec7c32c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db6x7" Nov 04 01:10:21.055469 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.055446 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" event={"ID":"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29","Type":"ContainerStarted","Data":"0484d558aa91301cc08167b8a00b236f9f612ae462154a0c1564a0cc797ceb1f"} Nov 04 01:10:21.055584 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.055487 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" event={"ID":"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29","Type":"ContainerStarted","Data":"83ffd8448ded9b9ec3ef6209f1bf5d4b96993fb173ab544e4321f7611f36e6ce"} Nov 04 01:10:21.057067 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.057049 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:10:21.057471 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.057449 2580 scope.go:122] "RemoveContainer" containerID="86f7eda2eef130111d4393a379809aa2517d0821e73d020723ac0d438c164361" Nov 04 01:10:21.057647 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:21.057624 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-8cfg7_openshift-console-operator(ae64edd1-43dc-406d-bcad-f7f34bb1f991)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" podUID="ae64edd1-43dc-406d-bcad-f7f34bb1f991" Nov 04 01:10:21.086252 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.086232 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vkk6b" Nov 04 01:10:21.093361 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.093343 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db6x7" Nov 04 01:10:21.294438 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.294342 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" podStartSLOduration=2.294327492 podStartE2EDuration="2.294327492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:21.175558956 +0000 UTC m=+45.851986602" watchObservedRunningTime="2026-11-04 01:10:21.294327492 +0000 UTC m=+45.970755143" Nov 04 01:10:21.470564 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.470532 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vkk6b"] Nov 04 01:10:21.477172 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:21.477147 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ddedc2c_735e_41e5_a5a9_3386ed890ee1.slice/crio-29855f357f0133564c0d8111980f94532023567cc7f628e722b906f68c1c6a2c WatchSource:0}: Error finding container 29855f357f0133564c0d8111980f94532023567cc7f628e722b906f68c1c6a2c: Status 404 returned error can't find the container with id 29855f357f0133564c0d8111980f94532023567cc7f628e722b906f68c1c6a2c Nov 04 01:10:21.488725 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.488696 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-db6x7"] Nov 04 01:10:21.494953 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:21.494926 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60eb5db_ec53_40bb_9a72_c712cec7c32c.slice/crio-c6c384c5ee28e82f224c188c97a0325dd539c534432b42c0bf543d8adaaa8a9e WatchSource:0}: Error finding container c6c384c5ee28e82f224c188c97a0325dd539c534432b42c0bf543d8adaaa8a9e: Status 404 returned error can't find the container with id c6c384c5ee28e82f224c188c97a0325dd539c534432b42c0bf543d8adaaa8a9e Nov 04 01:10:21.762524 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.762491 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-fk7k8"] Nov 04 01:10:21.765753 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.765734 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" Nov 04 01:10:21.819428 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.819382 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 04 01:10:21.819574 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.819384 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 04 01:10:21.819574 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.819495 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 04 01:10:21.819574 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.819509 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j2s6w\"" Nov 04 01:10:21.819574 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.819560 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 04 01:10:21.824878 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.824859 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-fk7k8"] Nov 04 01:10:21.853173 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.853150 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf-signing-key\") pod \"service-ca-5fffc97bdf-fk7k8\" (UID: \"c4093a5e-62e9-4f40-b3ad-15453e4f1bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" Nov 04 01:10:21.853263 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.853208 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chq5x\" (UniqueName: \"kubernetes.io/projected/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf-kube-api-access-chq5x\") pod \"service-ca-5fffc97bdf-fk7k8\" (UID: \"c4093a5e-62e9-4f40-b3ad-15453e4f1bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" Nov 04 01:10:21.853337 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.853319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf-signing-cabundle\") pod \"service-ca-5fffc97bdf-fk7k8\" (UID: \"c4093a5e-62e9-4f40-b3ad-15453e4f1bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" Nov 04 01:10:21.954322 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.954300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf-signing-cabundle\") pod \"service-ca-5fffc97bdf-fk7k8\" (UID: \"c4093a5e-62e9-4f40-b3ad-15453e4f1bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" Nov 04 01:10:21.954458 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.954406 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf-signing-key\") pod \"service-ca-5fffc97bdf-fk7k8\" (UID: \"c4093a5e-62e9-4f40-b3ad-15453e4f1bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" Nov 04 01:10:21.954516 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.954476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chq5x\" (UniqueName: \"kubernetes.io/projected/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf-kube-api-access-chq5x\") pod \"service-ca-5fffc97bdf-fk7k8\" (UID: \"c4093a5e-62e9-4f40-b3ad-15453e4f1bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" Nov 04 01:10:21.954997 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.954975 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf-signing-cabundle\") pod \"service-ca-5fffc97bdf-fk7k8\" (UID: \"c4093a5e-62e9-4f40-b3ad-15453e4f1bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" Nov 04 01:10:21.957482 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:21.957460 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf-signing-key\") pod \"service-ca-5fffc97bdf-fk7k8\" (UID: \"c4093a5e-62e9-4f40-b3ad-15453e4f1bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" Nov 04 01:10:22.016165 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:22.016123 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chq5x\" (UniqueName: \"kubernetes.io/projected/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf-kube-api-access-chq5x\") pod \"service-ca-5fffc97bdf-fk7k8\" (UID: \"c4093a5e-62e9-4f40-b3ad-15453e4f1bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" Nov 04 01:10:22.061429 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:22.061385 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db6x7" event={"ID":"c60eb5db-ec53-40bb-9a72-c712cec7c32c","Type":"ContainerStarted","Data":"c6c384c5ee28e82f224c188c97a0325dd539c534432b42c0bf543d8adaaa8a9e"} Nov 04 01:10:22.062454 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:22.062433 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vkk6b" event={"ID":"8ddedc2c-735e-41e5-a5a9-3386ed890ee1","Type":"ContainerStarted","Data":"29855f357f0133564c0d8111980f94532023567cc7f628e722b906f68c1c6a2c"} Nov 04 01:10:22.076751 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:22.076731 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" Nov 04 01:10:22.439602 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:22.439459 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-fk7k8"] Nov 04 01:10:22.446587 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:22.446554 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4093a5e_62e9_4f40_b3ad_15453e4f1bcf.slice/crio-9544ea47f6ea8e8786d664c14ef2178951957028bd8ff037e155ae0b8a532e82 WatchSource:0}: Error finding container 9544ea47f6ea8e8786d664c14ef2178951957028bd8ff037e155ae0b8a532e82: Status 404 returned error can't find the container with id 9544ea47f6ea8e8786d664c14ef2178951957028bd8ff037e155ae0b8a532e82 Nov 04 01:10:22.964950 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:22.964535 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:22.964950 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:22.964670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:22.965359 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:22.965337 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 01:10:22.965495 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:22.965481 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 01:10:22.965555 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:22.965532 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls podName:c5dd40be-e515-4fe8-be55-b93f5b97973e nodeName:}" failed. No retries permitted until 2026-11-04 01:10:26.965401609 +0000 UTC m=+51.641829258 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4j876" (UID: "c5dd40be-e515-4fe8-be55-b93f5b97973e") : secret "insights-runtime-extractor-tls" not found Nov 04 01:10:22.965555 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:22.965553 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm podName:c5dd40be-e515-4fe8-be55-b93f5b97973e nodeName:}" failed. No retries permitted until 2026-11-04 01:10:26.965542393 +0000 UTC m=+51.641970022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4j876" (UID: "c5dd40be-e515-4fe8-be55-b93f5b97973e") : configmap "kube-rbac-proxy" not found Nov 04 01:10:23.068220 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:23.068187 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" event={"ID":"c4093a5e-62e9-4f40-b3ad-15453e4f1bcf","Type":"ContainerStarted","Data":"12110ce6bc73bc6b464739927e6051c8a738ff668f509396498207dc8fa495e6"} Nov 04 01:10:23.068396 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:23.068279 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" event={"ID":"c4093a5e-62e9-4f40-b3ad-15453e4f1bcf","Type":"ContainerStarted","Data":"9544ea47f6ea8e8786d664c14ef2178951957028bd8ff037e155ae0b8a532e82"} Nov 04 01:10:23.118669 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:23.118641 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:23.118807 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:23.118700 2580 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:23.119106 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:23.119084 2580 scope.go:122] "RemoveContainer" containerID="86f7eda2eef130111d4393a379809aa2517d0821e73d020723ac0d438c164361" Nov 04 01:10:23.119340 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:23.119319 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-8cfg7_openshift-console-operator(ae64edd1-43dc-406d-bcad-f7f34bb1f991)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" podUID="ae64edd1-43dc-406d-bcad-f7f34bb1f991" Nov 04 01:10:23.185271 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:23.185225 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" podStartSLOduration=2.185211923 podStartE2EDuration="2.185211923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:23.183852093 +0000 UTC m=+47.860279740" watchObservedRunningTime="2026-11-04 01:10:23.185211923 +0000 UTC m=+47.861639566" Nov 04 01:10:24.067151 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:24.067120 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xsmlx_ffb55ed6-b225-4db5-a7fb-66a10868befc/dns-node-resolver/0.log" Nov 04 01:10:24.074735 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:24.074362 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vkk6b" event={"ID":"8ddedc2c-735e-41e5-a5a9-3386ed890ee1","Type":"ContainerStarted","Data":"9751b47d06ba9967d1525c46831272eadd6ad063443bd2c26437357534064e81"} Nov 04 01:10:24.076794 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:24.076767 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db6x7" event={"ID":"c60eb5db-ec53-40bb-9a72-c712cec7c32c","Type":"ContainerStarted","Data":"7a1dec7879dca59c6dc9904b34c6a576b8248ab2691868e982b91fcb86b9d861"} Nov 04 01:10:24.192186 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:24.192140 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vkk6b" podStartSLOduration=2.292073884 podStartE2EDuration="4.192126958s" totalImagesPullingTime="1.900053074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:24.1909071 +0000 UTC m=+48.867334747" watchObservedRunningTime="2026-11-04 01:10:24.192126958 +0000 UTC m=+48.868554632" Nov 04 01:10:24.308364 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:24.308319 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db6x7" podStartSLOduration=2.612358573 podStartE2EDuration="4.308306915s" totalImagesPullingTime="1.695948342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:24.306726737 +0000 UTC m=+48.983154386" watchObservedRunningTime="2026-11-04 01:10:24.308306915 +0000 UTC m=+48.984734562" Nov 04 01:10:25.065873 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:25.065841 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zzx84_0f50bc9b-91da-4d92-8553-c9f9476665e4/node-ca/0.log" Nov 04 01:10:25.466928 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:25.466897 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ms2pt_ca77e439-49cb-4ef0-9951-9564ab28ed3e/serve-healthcheck-canary/0.log" Nov 04 01:10:25.666707 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:25.666678 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-db6x7_c60eb5db-ec53-40bb-9a72-c712cec7c32c/migrator/0.log" Nov 04 01:10:25.867211 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:25.867153 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-vkk6b_8ddedc2c-735e-41e5-a5a9-3386ed890ee1/migrator/0.log" Nov 04 01:10:26.067458 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:26.067430 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-29mpf_b44a2a16-b00a-4113-b0ab-5cdcb955b325/kube-storage-version-migrator-operator/0.log" Nov 04 01:10:27.000615 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:27.000580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:27.000962 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:27.000687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:27.000962 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:27.000706 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 01:10:27.000962 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:27.000768 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm podName:c5dd40be-e515-4fe8-be55-b93f5b97973e nodeName:}" failed. No retries permitted until 2026-11-04 01:10:35.000750867 +0000 UTC m=+59.677178540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4j876" (UID: "c5dd40be-e515-4fe8-be55-b93f5b97973e") : configmap "kube-rbac-proxy" not found Nov 04 01:10:27.000962 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:27.000819 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 01:10:27.000962 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:27.000873 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls podName:c5dd40be-e515-4fe8-be55-b93f5b97973e nodeName:}" failed. No retries permitted until 2026-11-04 01:10:35.000861161 +0000 UTC m=+59.677288816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4j876" (UID: "c5dd40be-e515-4fe8-be55-b93f5b97973e") : secret "insights-runtime-extractor-tls" not found Nov 04 01:10:28.213013 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.212980 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:28.216341 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.216315 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:28.275575 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.275553 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:28.314632 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.314602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:28.314747 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.314653 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:28.314929 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.314901 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:28.315019 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:28.314934 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle podName:f2df8dc5-6403-4827-b6d6-9c209c595e05 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:44.314907752 +0000 UTC m=+68.991335397 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle") pod "router-default-f5c95ffcf-hl6wf" (UID: "f2df8dc5-6403-4827-b6d6-9c209c595e05") : configmap references non-existent config key: service-ca.crt Nov 04 01:10:28.316928 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.316903 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:28.317341 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.317325 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:28.402376 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.402351 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:28.416314 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.416291 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls\") pod \"image-registry-99c5d585-rzfqd\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:28.416439 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.416352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:28.416502 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:28.416436 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 01:10:28.416502 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:28.416452 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-99c5d585-rzfqd: secret "image-registry-tls" not found Nov 04 01:10:28.416502 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.416478 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-582gz\" (UID: \"5b8947a9-b2f0-4de2-a1b8-58342b6b3240\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" Nov 04 01:10:28.416502 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:28.416496 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls podName:a99591bd-c354-4d6c-940d-51b0969ee916 nodeName:}" failed. No retries permitted until 2026-11-04 01:10:44.416482593 +0000 UTC m=+69.092910242 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls") pod "image-registry-99c5d585-rzfqd" (UID: "a99591bd-c354-4d6c-940d-51b0969ee916") : secret "image-registry-tls" not found Nov 04 01:10:28.416695 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:28.416525 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 01:10:28.416695 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:28.416544 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6556fd5cf6-2dl7j: secret "image-registry-tls" not found Nov 04 01:10:28.416695 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:28.416600 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls podName:c2aa236a-4ff4-4b39-90ff-c782721444ee nodeName:}" failed. No retries permitted until 2026-11-04 01:10:44.416583841 +0000 UTC m=+69.093011483 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls") pod "image-registry-6556fd5cf6-2dl7j" (UID: "c2aa236a-4ff4-4b39-90ff-c782721444ee") : secret "image-registry-tls" not found Nov 04 01:10:28.418689 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.418671 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-582gz\" (UID: \"5b8947a9-b2f0-4de2-a1b8-58342b6b3240\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" Nov 04 01:10:28.455976 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.455956 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" Nov 04 01:10:28.518017 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.517988 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:28.519959 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.519938 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:28.552485 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.552464 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:28.645785 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.645761 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl"] Nov 04 01:10:28.652187 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:28.652160 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c5b017_8f43_41dc_b79b_0353de290289.slice/crio-0750204d2e8b59273e5d36bb112a2603bbe424206ef65dd634bb6a7a3efa927d WatchSource:0}: Error finding container 0750204d2e8b59273e5d36bb112a2603bbe424206ef65dd634bb6a7a3efa927d: Status 404 returned error can't find the container with id 0750204d2e8b59273e5d36bb112a2603bbe424206ef65dd634bb6a7a3efa927d Nov 04 01:10:28.765569 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.765520 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k"] Nov 04 01:10:28.818922 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.818901 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-582gz"] Nov 04 01:10:28.824856 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:28.824834 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8947a9_b2f0_4de2_a1b8_58342b6b3240.slice/crio-76b3d30e17580e47f2e408a0debe8067481a8f8432dc342f8f86a0b6cc06020d WatchSource:0}: Error finding container 76b3d30e17580e47f2e408a0debe8067481a8f8432dc342f8f86a0b6cc06020d: Status 404 returned error can't find the container with id 76b3d30e17580e47f2e408a0debe8067481a8f8432dc342f8f86a0b6cc06020d Nov 04 01:10:28.918039 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:28.918018 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2h8lq"] Nov 04 01:10:28.923782 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:28.923760 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96cde4e_0238_48e6_b343_a0592f6e159e.slice/crio-1f80bab462e52ddc7607885b1de473b4caed00db64ad6f17487d337795134196 WatchSource:0}: Error finding container 1f80bab462e52ddc7607885b1de473b4caed00db64ad6f17487d337795134196: Status 404 returned error can't find the container with id 1f80bab462e52ddc7607885b1de473b4caed00db64ad6f17487d337795134196 Nov 04 01:10:29.091151 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:29.091091 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2h8lq" event={"ID":"e96cde4e-0238-48e6-b343-a0592f6e159e","Type":"ContainerStarted","Data":"1f80bab462e52ddc7607885b1de473b4caed00db64ad6f17487d337795134196"} Nov 04 01:10:29.092166 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:29.092142 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" event={"ID":"ef7aa61c-2789-49ed-989d-fa77fed0ff0f","Type":"ContainerStarted","Data":"30e0df48fcd09cdce51e26071a7daec8132477b4ae20bb7af8f568959ad13341"} Nov 04 01:10:29.093235 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:29.093219 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" event={"ID":"5b8947a9-b2f0-4de2-a1b8-58342b6b3240","Type":"ContainerStarted","Data":"76b3d30e17580e47f2e408a0debe8067481a8f8432dc342f8f86a0b6cc06020d"} Nov 04 01:10:29.094179 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:29.094161 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" event={"ID":"75c5b017-8f43-41dc-b79b-0353de290289","Type":"ContainerStarted","Data":"0750204d2e8b59273e5d36bb112a2603bbe424206ef65dd634bb6a7a3efa927d"} Nov 04 01:10:30.822055 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:30.821649 2580 scope.go:122] "RemoveContainer" containerID="86f7eda2eef130111d4393a379809aa2517d0821e73d020723ac0d438c164361" Nov 04 01:10:31.108714 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:31.107492 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:10:31.108714 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:31.107630 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" event={"ID":"ae64edd1-43dc-406d-bcad-f7f34bb1f991","Type":"ContainerStarted","Data":"5faf707489c320700eb0fa3e5cc95cb6dced9ab29cb5586fea5ab6863d8617f1"} Nov 04 01:10:31.108714 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:31.108485 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:31.109967 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:31.109932 2580 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-8cfg7 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.22:8443/readyz\": dial tcp 10.132.0.22:8443: connect: connection refused" start-of-body= Nov 04 01:10:31.110068 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:31.109989 2580 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" podUID="ae64edd1-43dc-406d-bcad-f7f34bb1f991" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.22:8443/readyz\": dial tcp 10.132.0.22:8443: connect: connection refused" Nov 04 01:10:31.110579 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:31.110554 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" event={"ID":"5b8947a9-b2f0-4de2-a1b8-58342b6b3240","Type":"ContainerStarted","Data":"7f08370466229eabb5bc26e163db7b7cc12cb62e24a0e3b693b77a376e3e914a"} Nov 04 01:10:31.250442 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:31.245837 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" podStartSLOduration=435.416932987 podStartE2EDuration="7m20.245817696s" totalImagesPullingTime="4.828884709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:31.243869795 +0000 UTC m=+55.920297443" watchObservedRunningTime="2026-11-04 01:10:31.245817696 +0000 UTC m=+55.922245345" Nov 04 01:10:31.362888 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:31.362763 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" podStartSLOduration=405.194459154 podStartE2EDuration="6m47.362745089s" totalImagesPullingTime="2.168285935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:31.361972756 +0000 UTC m=+56.038400634" watchObservedRunningTime="2026-11-04 01:10:31.362745089 +0000 UTC m=+56.039172738" Nov 04 01:10:32.115461 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:32.115395 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" event={"ID":"75c5b017-8f43-41dc-b79b-0353de290289","Type":"ContainerStarted","Data":"0cbea2f49434355fc92efe50d65c818cc55d575dab552178f5ab2e5e3d8ccc18"} Nov 04 01:10:32.117222 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:32.117196 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2h8lq" event={"ID":"e96cde4e-0238-48e6-b343-a0592f6e159e","Type":"ContainerStarted","Data":"59b9266e25e1339649bec17f58f56fca19465720492ac8787fbab34df0fbb54c"} Nov 04 01:10:32.117333 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:32.117247 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:32.117333 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:32.117270 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2h8lq" event={"ID":"e96cde4e-0238-48e6-b343-a0592f6e159e","Type":"ContainerStarted","Data":"ad4c381a407bc5bb25fc2395670be4a92cc1b335d5e5735ef261f4a16e836321"} Nov 04 01:10:32.118936 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:32.118904 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" event={"ID":"ef7aa61c-2789-49ed-989d-fa77fed0ff0f","Type":"ContainerStarted","Data":"7ba87ea2e40a71c9c8fdb65d82f214cdb9abbe2ffa0d2d822a926683fcad269c"} Nov 04 01:10:32.119042 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:32.118951 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" event={"ID":"ef7aa61c-2789-49ed-989d-fa77fed0ff0f","Type":"ContainerStarted","Data":"147b0383748b0242e65c05daba143c3a9d7a930e5b4666bba7cd02c878808b68"} Nov 04 01:10:32.123107 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:32.123083 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:32.274497 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:32.274442 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" podStartSLOduration=438.163663447 podStartE2EDuration="7m21.274424672s" totalImagesPullingTime="3.110761225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:32.272911325 +0000 UTC m=+56.949338973" watchObservedRunningTime="2026-11-04 01:10:32.274424672 +0000 UTC m=+56.950852317" Nov 04 01:10:32.509835 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:32.509780 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" podStartSLOduration=438.480783937 podStartE2EDuration="7m21.509766999s" totalImagesPullingTime="3.028983062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:32.508601977 +0000 UTC m=+57.185029623" watchObservedRunningTime="2026-11-04 01:10:32.509766999 +0000 UTC m=+57.186194646" Nov 04 01:10:32.627320 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:32.627270 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-2h8lq" podStartSLOduration=17.654492862 podStartE2EDuration="20.627255328s" totalImagesPullingTime="2.972762466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:32.626598515 +0000 UTC m=+57.303026166" watchObservedRunningTime="2026-11-04 01:10:32.627255328 +0000 UTC m=+57.303682976" Nov 04 01:10:35.080069 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:35.080031 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:35.080473 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:35.080111 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:35.080584 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:35.080564 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:35.082298 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:35.082274 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:35.259480 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:35.259437 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:35.626019 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:35.625988 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4j876"] Nov 04 01:10:35.663315 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:35.663287 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5dd40be_e515_4fe8_be55_b93f5b97973e.slice/crio-f6fbf21c0ddf595ca72a79c5e68ff7f45cd411422580b519df444546ce79f4f6 WatchSource:0}: Error finding container f6fbf21c0ddf595ca72a79c5e68ff7f45cd411422580b519df444546ce79f4f6: Status 404 returned error can't find the container with id f6fbf21c0ddf595ca72a79c5e68ff7f45cd411422580b519df444546ce79f4f6 Nov 04 01:10:35.995232 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:35.995208 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:36.133636 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:36.133604 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4j876" event={"ID":"c5dd40be-e515-4fe8-be55-b93f5b97973e","Type":"ContainerStarted","Data":"20e06b5a77660c891c754f0f2e75a298cd90641b80f6568bcf8746b9db227e49"} Nov 04 01:10:36.133975 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:36.133655 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4j876" event={"ID":"c5dd40be-e515-4fe8-be55-b93f5b97973e","Type":"ContainerStarted","Data":"f6fbf21c0ddf595ca72a79c5e68ff7f45cd411422580b519df444546ce79f4f6"} Nov 04 01:10:37.138877 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:37.138853 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4j876" event={"ID":"c5dd40be-e515-4fe8-be55-b93f5b97973e","Type":"ContainerStarted","Data":"fea01fe6d6c4b6ad76099cf9bcbb6fdaed471cc03213c613a3d72c9c874c4c84"} Nov 04 01:10:40.155631 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.155585 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4j876" event={"ID":"c5dd40be-e515-4fe8-be55-b93f5b97973e","Type":"ContainerStarted","Data":"3fa50120c9607beafcd62d3bc18429ab18be0a7c176840eec0186009079b02d2"} Nov 04 01:10:40.275133 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.275074 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4j876" podStartSLOduration=17.696585714 podStartE2EDuration="21.27506105s" totalImagesPullingTime="3.578475336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:10:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:40.274063065 +0000 UTC m=+64.950490712" watchObservedRunningTime="2026-11-04 01:10:40.27506105 +0000 UTC m=+64.951488698" Nov 04 01:10:40.489220 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.489190 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-47hlt"] Nov 04 01:10:40.491404 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.491383 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-47hlt" Nov 04 01:10:40.566948 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.566921 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 04 01:10:40.566948 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.566925 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lwspx\"" Nov 04 01:10:40.578107 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.578083 2580 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-99c5d585-rzfqd"] Nov 04 01:10:40.578309 ip-10-0-138-179 kubenswrapper[2580]: E1104 01:10:40.578292 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-99c5d585-rzfqd" podUID="a99591bd-c354-4d6c-940d-51b0969ee916" Nov 04 01:10:40.599327 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.599300 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-gsvcf"] Nov 04 01:10:40.601650 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.601634 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-gsvcf" Nov 04 01:10:40.620733 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.620710 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-47hlt"] Nov 04 01:10:40.628476 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.628455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/747ed7ed-d05b-434a-8360-acc2292c2257-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-47hlt\" (UID: \"747ed7ed-d05b-434a-8360-acc2292c2257\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-47hlt" Nov 04 01:10:40.662763 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.662739 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 04 01:10:40.662976 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.662835 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 04 01:10:40.663049 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.662900 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-55klw\"" Nov 04 01:10:40.663847 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.663826 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-gsvcf"] Nov 04 01:10:40.689286 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.689260 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69bf8d55f8-x6rb7"] Nov 04 01:10:40.691678 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.691661 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.729181 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.729158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d1de04c-5111-415b-bdcc-20dfa9d0e6f1-tmp\") pod \"downloads-6dd789c8b7-gsvcf\" (UID: \"5d1de04c-5111-415b-bdcc-20dfa9d0e6f1\") " pod="openshift-console/downloads-6dd789c8b7-gsvcf" Nov 04 01:10:40.729263 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.729190 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rgj\" (UniqueName: \"kubernetes.io/projected/5d1de04c-5111-415b-bdcc-20dfa9d0e6f1-kube-api-access-j8rgj\") pod \"downloads-6dd789c8b7-gsvcf\" (UID: \"5d1de04c-5111-415b-bdcc-20dfa9d0e6f1\") " pod="openshift-console/downloads-6dd789c8b7-gsvcf" Nov 04 01:10:40.729374 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.729356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/747ed7ed-d05b-434a-8360-acc2292c2257-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-47hlt\" (UID: \"747ed7ed-d05b-434a-8360-acc2292c2257\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-47hlt" Nov 04 01:10:40.731637 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.731618 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/747ed7ed-d05b-434a-8360-acc2292c2257-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-47hlt\" (UID: \"747ed7ed-d05b-434a-8360-acc2292c2257\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-47hlt" Nov 04 01:10:40.751230 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.751183 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69bf8d55f8-x6rb7"] Nov 04 01:10:40.807196 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.807167 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-47hlt" Nov 04 01:10:40.830377 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.830350 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-registry-certificates\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.830505 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.830397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-installation-pull-secrets\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.830558 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.830509 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d1de04c-5111-415b-bdcc-20dfa9d0e6f1-tmp\") pod \"downloads-6dd789c8b7-gsvcf\" (UID: \"5d1de04c-5111-415b-bdcc-20dfa9d0e6f1\") " pod="openshift-console/downloads-6dd789c8b7-gsvcf" Nov 04 01:10:40.830558 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.830536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rgj\" (UniqueName: \"kubernetes.io/projected/5d1de04c-5111-415b-bdcc-20dfa9d0e6f1-kube-api-access-j8rgj\") pod \"downloads-6dd789c8b7-gsvcf\" (UID: \"5d1de04c-5111-415b-bdcc-20dfa9d0e6f1\") " pod="openshift-console/downloads-6dd789c8b7-gsvcf" Nov 04 01:10:40.830650 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.830558 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-image-registry-private-configuration\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.830650 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.830581 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlvzh\" (UniqueName: \"kubernetes.io/projected/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-kube-api-access-xlvzh\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.830739 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.830713 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-trusted-ca\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.830782 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.830745 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-ca-trust-extracted\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.830782 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.830769 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-bound-sa-token\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.830879 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.830793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-registry-tls\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.831002 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.830979 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d1de04c-5111-415b-bdcc-20dfa9d0e6f1-tmp\") pod \"downloads-6dd789c8b7-gsvcf\" (UID: \"5d1de04c-5111-415b-bdcc-20dfa9d0e6f1\") " pod="openshift-console/downloads-6dd789c8b7-gsvcf" Nov 04 01:10:40.902298 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.902261 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rgj\" (UniqueName: \"kubernetes.io/projected/5d1de04c-5111-415b-bdcc-20dfa9d0e6f1-kube-api-access-j8rgj\") pod \"downloads-6dd789c8b7-gsvcf\" (UID: \"5d1de04c-5111-415b-bdcc-20dfa9d0e6f1\") " pod="openshift-console/downloads-6dd789c8b7-gsvcf" Nov 04 01:10:40.911988 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.911962 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-gsvcf" Nov 04 01:10:40.931709 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.931683 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-trusted-ca\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.931845 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.931714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-ca-trust-extracted\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.931845 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.931735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-bound-sa-token\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.931948 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.931867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-registry-tls\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.931948 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.931934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-registry-certificates\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.932041 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.931992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-installation-pull-secrets\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.932096 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.932082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-image-registry-private-configuration\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.932145 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.932113 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlvzh\" (UniqueName: \"kubernetes.io/projected/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-kube-api-access-xlvzh\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.932145 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.932124 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-ca-trust-extracted\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.932679 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.932657 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-trusted-ca\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.932922 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.932899 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-registry-certificates\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.934667 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.934640 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-registry-tls\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.934911 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.934881 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-image-registry-private-configuration\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.935007 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.934945 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-installation-pull-secrets\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.992981 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.992946 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-bound-sa-token\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:40.992981 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:40.992980 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlvzh\" (UniqueName: \"kubernetes.io/projected/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-kube-api-access-xlvzh\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:41.053953 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.053876 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s46xs\"" Nov 04 01:10:41.062101 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.062044 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:41.159677 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.159651 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:41.166902 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.166880 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:41.171339 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.171319 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-47hlt"] Nov 04 01:10:41.177562 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:41.177539 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747ed7ed_d05b_434a_8360_acc2292c2257.slice/crio-a889499cabdba0d2fd9a2ef3f5313b44a484207a29a7b6f6f795fcf287ea34be WatchSource:0}: Error finding container a889499cabdba0d2fd9a2ef3f5313b44a484207a29a7b6f6f795fcf287ea34be: Status 404 returned error can't find the container with id a889499cabdba0d2fd9a2ef3f5313b44a484207a29a7b6f6f795fcf287ea34be Nov 04 01:10:41.234248 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.234206 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a99591bd-c354-4d6c-940d-51b0969ee916-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a99591bd-c354-4d6c-940d-51b0969ee916-trusted-ca\") pod \"a99591bd-c354-4d6c-940d-51b0969ee916\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " Nov 04 01:10:41.234347 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.234272 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a99591bd-c354-4d6c-940d-51b0969ee916-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a99591bd-c354-4d6c-940d-51b0969ee916-ca-trust-extracted\") pod \"a99591bd-c354-4d6c-940d-51b0969ee916\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " Nov 04 01:10:41.234347 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.234306 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-bound-sa-token\") pod \"a99591bd-c354-4d6c-940d-51b0969ee916\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " Nov 04 01:10:41.234480 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.234370 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a99591bd-c354-4d6c-940d-51b0969ee916-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a99591bd-c354-4d6c-940d-51b0969ee916-installation-pull-secrets\") pod \"a99591bd-c354-4d6c-940d-51b0969ee916\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " Nov 04 01:10:41.234480 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.234409 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a99591bd-c354-4d6c-940d-51b0969ee916-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a99591bd-c354-4d6c-940d-51b0969ee916-image-registry-private-configuration\") pod \"a99591bd-c354-4d6c-940d-51b0969ee916\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " Nov 04 01:10:41.234579 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.234478 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a99591bd-c354-4d6c-940d-51b0969ee916-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a99591bd-c354-4d6c-940d-51b0969ee916-registry-certificates\") pod \"a99591bd-c354-4d6c-940d-51b0969ee916\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " Nov 04 01:10:41.234579 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.234509 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-kube-api-access-p5dhv\" (UniqueName: \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-kube-api-access-p5dhv\") pod \"a99591bd-c354-4d6c-940d-51b0969ee916\" (UID: \"a99591bd-c354-4d6c-940d-51b0969ee916\") " Nov 04 01:10:41.234672 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.234584 2580 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99591bd-c354-4d6c-940d-51b0969ee916-ca-trust-extracted" pod "a99591bd-c354-4d6c-940d-51b0969ee916" (UID: "a99591bd-c354-4d6c-940d-51b0969ee916"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:10:41.234721 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.234662 2580 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99591bd-c354-4d6c-940d-51b0969ee916-trusted-ca" pod "a99591bd-c354-4d6c-940d-51b0969ee916" (UID: "a99591bd-c354-4d6c-940d-51b0969ee916"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:10:41.234878 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.234853 2580 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99591bd-c354-4d6c-940d-51b0969ee916-registry-certificates" pod "a99591bd-c354-4d6c-940d-51b0969ee916" (UID: "a99591bd-c354-4d6c-940d-51b0969ee916"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:10:41.235038 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.235018 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a99591bd-c354-4d6c-940d-51b0969ee916-ca-trust-extracted\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:10:41.235112 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.235039 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a99591bd-c354-4d6c-940d-51b0969ee916-registry-certificates\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:10:41.235112 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.235051 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a99591bd-c354-4d6c-940d-51b0969ee916-trusted-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:10:41.236701 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.236679 2580 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99591bd-c354-4d6c-940d-51b0969ee916-installation-pull-secrets" pod "a99591bd-c354-4d6c-940d-51b0969ee916" (UID: "a99591bd-c354-4d6c-940d-51b0969ee916"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:10:41.236773 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.236724 2580 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-kube-api-access-p5dhv" pod "a99591bd-c354-4d6c-940d-51b0969ee916" (UID: "a99591bd-c354-4d6c-940d-51b0969ee916"). InnerVolumeSpecName "kube-api-access-p5dhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:10:41.236918 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.236893 2580 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99591bd-c354-4d6c-940d-51b0969ee916-image-registry-private-configuration" pod "a99591bd-c354-4d6c-940d-51b0969ee916" (UID: "a99591bd-c354-4d6c-940d-51b0969ee916"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:10:41.236918 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.236905 2580 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-bound-sa-token" pod "a99591bd-c354-4d6c-940d-51b0969ee916" (UID: "a99591bd-c354-4d6c-940d-51b0969ee916"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:10:41.280516 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.280486 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-gsvcf"] Nov 04 01:10:41.286786 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:41.286763 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1de04c_5111_415b_bdcc_20dfa9d0e6f1.slice/crio-5ecbf13bab42e195b5371678969094ac07ed75be0035a7afed257df0aadbe0af WatchSource:0}: Error finding container 5ecbf13bab42e195b5371678969094ac07ed75be0035a7afed257df0aadbe0af: Status 404 returned error can't find the container with id 5ecbf13bab42e195b5371678969094ac07ed75be0035a7afed257df0aadbe0af Nov 04 01:10:41.336164 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.336098 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a99591bd-c354-4d6c-940d-51b0969ee916-installation-pull-secrets\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:10:41.336164 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.336124 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a99591bd-c354-4d6c-940d-51b0969ee916-image-registry-private-configuration\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:10:41.336164 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.336140 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5dhv\" (UniqueName: \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-kube-api-access-p5dhv\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:10:41.336164 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.336154 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-bound-sa-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:10:41.430666 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.430635 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69bf8d55f8-x6rb7"] Nov 04 01:10:41.437943 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:41.437919 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81014c6f_b8a1_4f60_abb7_52cfb2c2f9fe.slice/crio-8c35ab28fb00962801a8fe34285bf6942fc29ad7360a9e7604f957437f7d950f WatchSource:0}: Error finding container 8c35ab28fb00962801a8fe34285bf6942fc29ad7360a9e7604f957437f7d950f: Status 404 returned error can't find the container with id 8c35ab28fb00962801a8fe34285bf6942fc29ad7360a9e7604f957437f7d950f Nov 04 01:10:41.739845 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.739809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:41.740028 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.739959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs\") pod \"network-metrics-daemon-zfj7b\" (UID: \"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8\") " pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:10:41.792191 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.792166 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 01:10:41.792331 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.792165 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 01:10:41.802642 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.802619 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:41.802642 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.802631 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs\") pod \"network-metrics-daemon-zfj7b\" (UID: \"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8\") " pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:10:41.840631 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.840602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxlfx\" (UniqueName: \"kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx\") pod \"network-check-target-xqtpx\" (UID: \"42f02121-70f2-43d1-be9d-2a81425ec97d\") " pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:10:41.842725 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.842706 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxlfx\" (UniqueName: \"kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx\") pod \"network-check-target-xqtpx\" (UID: \"42f02121-70f2-43d1-be9d-2a81425ec97d\") " pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:10:41.862693 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.862673 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-8hhs6\"" Nov 04 01:10:41.870791 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:41.870764 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:42.112743 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:42.112665 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-x4nsn\"" Nov 04 01:10:42.121377 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:42.121348 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:10:42.124406 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:42.124373 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:42.149481 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:42.149456 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8lz6w\"" Nov 04 01:10:42.157650 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:42.157624 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:10:42.164593 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:42.164563 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" event={"ID":"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe","Type":"ContainerStarted","Data":"c0c9ae8c535140f8889c9e00b4b3726dd4490c1d2d24d8a4f72dd0ec5d66bcfd"} Nov 04 01:10:42.164955 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:42.164921 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" event={"ID":"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe","Type":"ContainerStarted","Data":"8c35ab28fb00962801a8fe34285bf6942fc29ad7360a9e7604f957437f7d950f"} Nov 04 01:10:42.168862 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:42.168270 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-gsvcf" event={"ID":"5d1de04c-5111-415b-bdcc-20dfa9d0e6f1","Type":"ContainerStarted","Data":"5ecbf13bab42e195b5371678969094ac07ed75be0035a7afed257df0aadbe0af"} Nov 04 01:10:42.173064 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:42.173037 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-47hlt" event={"ID":"747ed7ed-d05b-434a-8360-acc2292c2257","Type":"ContainerStarted","Data":"a889499cabdba0d2fd9a2ef3f5313b44a484207a29a7b6f6f795fcf287ea34be"} Nov 04 01:10:42.173197 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:42.173090 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-99c5d585-rzfqd" Nov 04 01:10:42.234613 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:42.234546 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b2jgm"] Nov 04 01:10:42.422509 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:42.422434 2580 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-99c5d585-rzfqd"] Nov 04 01:10:42.426269 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:42.426219 2580 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-99c5d585-rzfqd"] Nov 04 01:10:42.488099 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:42.488065 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xqtpx"] Nov 04 01:10:42.502471 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:42.496048 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f02121_70f2_43d1_be9d_2a81425ec97d.slice/crio-875c7ad36433ca2510d1f9bc776d1abf6a9ee2bef42bae468edd1159927d291a WatchSource:0}: Error finding container 875c7ad36433ca2510d1f9bc776d1abf6a9ee2bef42bae468edd1159927d291a: Status 404 returned error can't find the container with id 875c7ad36433ca2510d1f9bc776d1abf6a9ee2bef42bae468edd1159927d291a Nov 04 01:10:42.539571 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:42.539490 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zfj7b"] Nov 04 01:10:42.547370 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:42.546867 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2eb46c_efdf_48fc_aaf9_78acb8aaaac8.slice/crio-69a81c3f70096360826bcc68061e4a63af85de5c099c0524fa9ba022f6671d82 WatchSource:0}: Error finding container 69a81c3f70096360826bcc68061e4a63af85de5c099c0524fa9ba022f6671d82: Status 404 returned error can't find the container with id 69a81c3f70096360826bcc68061e4a63af85de5c099c0524fa9ba022f6671d82 Nov 04 01:10:42.547370 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:42.547332 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a99591bd-c354-4d6c-940d-51b0969ee916-registry-tls\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:10:42.549044 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:42.548942 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" podStartSLOduration=2.548902478 podStartE2EDuration="2.548902478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:42.547115906 +0000 UTC m=+67.223543557" watchObservedRunningTime="2026-11-04 01:10:42.548902478 +0000 UTC m=+67.225330125" Nov 04 01:10:43.179598 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:43.179507 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-47hlt" event={"ID":"747ed7ed-d05b-434a-8360-acc2292c2257","Type":"ContainerStarted","Data":"bbdcaabb9615fe0236030d0ed5d336a276184e52454a71bd539b71437a81a035"} Nov 04 01:10:43.180027 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:43.179804 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-47hlt" Nov 04 01:10:43.181507 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:43.181478 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zfj7b" event={"ID":"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8","Type":"ContainerStarted","Data":"69a81c3f70096360826bcc68061e4a63af85de5c099c0524fa9ba022f6671d82"} Nov 04 01:10:43.183373 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:43.183349 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xqtpx" event={"ID":"42f02121-70f2-43d1-be9d-2a81425ec97d","Type":"ContainerStarted","Data":"0ee6405c916c33025a7aa2a04aa3955acc92dd4c39a5ca546a18beb74933e64a"} Nov 04 01:10:43.183490 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:43.183398 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:10:43.183490 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:43.183445 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xqtpx" event={"ID":"42f02121-70f2-43d1-be9d-2a81425ec97d","Type":"ContainerStarted","Data":"875c7ad36433ca2510d1f9bc776d1abf6a9ee2bef42bae468edd1159927d291a"} Nov 04 01:10:43.184961 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:43.184937 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b2jgm" event={"ID":"3ae06262-6e59-4775-bedc-5305ace1fae6","Type":"ContainerStarted","Data":"b7f25d5d02c71d60f4299230aed43c4c7113dfb0a34777bf62d08ea66c862290"} Nov 04 01:10:43.185073 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:43.184979 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:43.185359 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:43.185341 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-47hlt" Nov 04 01:10:43.296693 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:43.296636 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-47hlt" podStartSLOduration=1.641079239 podStartE2EDuration="3.296618434s" totalImagesPullingTime="1.655539195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:43.295637125 +0000 UTC m=+67.972064772" watchObservedRunningTime="2026-11-04 01:10:43.296618434 +0000 UTC m=+67.973046083" Nov 04 01:10:43.829233 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:43.829162 2580 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a99591bd-c354-4d6c-940d-51b0969ee916" path="/var/lib/kubelet/pods/a99591bd-c354-4d6c-940d-51b0969ee916/volumes" Nov 04 01:10:44.365072 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:44.365030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:44.365782 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:44.365763 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:44.466559 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:44.466515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:44.474028 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:44.473996 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:44.565494 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:44.565398 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-f6ftn\"" Nov 04 01:10:44.573809 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:44.573751 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:44.685229 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:44.684332 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:45.072261 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:45.072207 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xqtpx" podStartSLOduration=69.072193949 podStartE2EDuration="1m9.072193949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:43.58127433 +0000 UTC m=+68.257701976" watchObservedRunningTime="2026-11-04 01:10:45.072193949 +0000 UTC m=+69.748621594" Nov 04 01:10:45.073923 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:45.073897 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f5c95ffcf-hl6wf"] Nov 04 01:10:45.080982 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:45.080954 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2df8dc5_6403_4827_b6d6_9c209c595e05.slice/crio-65ee99eef508daea64d5d7e15539739e09ad12a1e442f4067af4dc52aabf6ba0 WatchSource:0}: Error finding container 65ee99eef508daea64d5d7e15539739e09ad12a1e442f4067af4dc52aabf6ba0: Status 404 returned error can't find the container with id 65ee99eef508daea64d5d7e15539739e09ad12a1e442f4067af4dc52aabf6ba0 Nov 04 01:10:45.130485 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:45.130460 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6556fd5cf6-2dl7j"] Nov 04 01:10:45.132573 ip-10-0-138-179 kubenswrapper[2580]: W1104 01:10:45.132548 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2aa236a_4ff4_4b39_90ff_c782721444ee.slice/crio-44d7b0a2497bef282b17ba2e0e4b41b6cca26c8d3a27b5de2f18c3746254701d WatchSource:0}: Error finding container 44d7b0a2497bef282b17ba2e0e4b41b6cca26c8d3a27b5de2f18c3746254701d: Status 404 returned error can't find the container with id 44d7b0a2497bef282b17ba2e0e4b41b6cca26c8d3a27b5de2f18c3746254701d Nov 04 01:10:45.197100 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:45.196665 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zfj7b" event={"ID":"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8","Type":"ContainerStarted","Data":"69780a093141cd1d486fe1c9fc864280f7b75ceed37cdb5ced76674c14e2c2d5"} Nov 04 01:10:45.197100 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:45.196731 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zfj7b" event={"ID":"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8","Type":"ContainerStarted","Data":"67c29f9d50c6a35e5492889d1cdc3a00f77104c70dbe180bb1eb0f12e2ea3016"} Nov 04 01:10:45.198631 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:45.198568 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" event={"ID":"c2aa236a-4ff4-4b39-90ff-c782721444ee","Type":"ContainerStarted","Data":"44d7b0a2497bef282b17ba2e0e4b41b6cca26c8d3a27b5de2f18c3746254701d"} Nov 04 01:10:45.201179 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:45.201098 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" event={"ID":"f2df8dc5-6403-4827-b6d6-9c209c595e05","Type":"ContainerStarted","Data":"0157af4e5f92c0719fd57da69030593c4e9740b282d72ccd6cf4c4f3e856c5f6"} Nov 04 01:10:45.201179 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:45.201137 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" event={"ID":"f2df8dc5-6403-4827-b6d6-9c209c595e05","Type":"ContainerStarted","Data":"65ee99eef508daea64d5d7e15539739e09ad12a1e442f4067af4dc52aabf6ba0"} Nov 04 01:10:45.314432 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:45.314372 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zfj7b" podStartSLOduration=67.440364404 podStartE2EDuration="1m9.314357671s" totalImagesPullingTime="1.873993267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:45.313105276 +0000 UTC m=+69.989532924" watchObservedRunningTime="2026-11-04 01:10:45.314357671 +0000 UTC m=+69.990785316" Nov 04 01:10:45.435933 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:45.435846 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" podStartSLOduration=454.435831975 podStartE2EDuration="7m34.435831975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:45.434270351 +0000 UTC m=+70.110698000" watchObservedRunningTime="2026-11-04 01:10:45.435831975 +0000 UTC m=+70.112259619" Nov 04 01:10:45.574841 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:45.574806 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:45.578127 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:45.578094 2580 patch_prober.go:29] interesting pod/router-default-f5c95ffcf-hl6wf container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Nov 04 01:10:45.578127 ip-10-0-138-179 kubenswrapper[2580]: [-]has-synced failed: reason withheld Nov 04 01:10:45.578127 ip-10-0-138-179 kubenswrapper[2580]: [+]process-running ok Nov 04 01:10:45.578127 ip-10-0-138-179 kubenswrapper[2580]: healthz check failed Nov 04 01:10:45.578385 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:45.578156 2580 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" podUID="f2df8dc5-6403-4827-b6d6-9c209c595e05" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Nov 04 01:10:46.207513 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:46.207480 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" event={"ID":"c2aa236a-4ff4-4b39-90ff-c782721444ee","Type":"ContainerStarted","Data":"e1b61d649bd932d80eca679d54af7f80e16c71675a22d6b846005bd1d901c3ec"} Nov 04 01:10:46.207751 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:46.207732 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:46.332758 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:46.332707 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" podStartSLOduration=61.332692314 podStartE2EDuration="1m1.332692314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:46.331234348 +0000 UTC m=+71.007662001" watchObservedRunningTime="2026-11-04 01:10:46.332692314 +0000 UTC m=+71.009119960" Nov 04 01:10:46.576050 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:46.575981 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:47.212808 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:47.212449 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:47.213886 ip-10-0-138-179 kubenswrapper[2580]: I1104 01:10:47.213855 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:49.918308 ip-10-0-138-179 systemd[1]: Stopping Kubernetes Kubelet... Nov 04 01:10:49.927078 ip-10-0-138-179 systemd[1]: kubelet.service: Deactivated successfully. Nov 04 01:10:49.927261 ip-10-0-138-179 systemd[1]: Stopped Kubernetes Kubelet. Nov 04 01:10:49.927497 ip-10-0-138-179 systemd[1]: kubelet.service: Consumed 4.615s CPU time. Nov 04 01:10:49.936136 ip-10-0-138-179 systemd[1]: Starting Kubernetes Kubelet... Nov 04 01:10:50.024840 ip-10-0-138-179 kubenswrapper[5995]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 01:10:50.024840 ip-10-0-138-179 kubenswrapper[5995]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 01:10:50.024840 ip-10-0-138-179 kubenswrapper[5995]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 01:10:50.024840 ip-10-0-138-179 kubenswrapper[5995]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 01:10:50.026879 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026792 5995 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:10:50.026879 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026879 5995 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026884 5995 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026889 5995 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026893 5995 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026896 5995 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026899 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026902 5995 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026904 5995 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026907 5995 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026910 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026913 5995 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026915 5995 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026918 5995 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026926 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026929 5995 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026931 5995 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026934 5995 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026937 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026940 5995 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:10:50.026940 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026943 5995 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026945 5995 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026948 5995 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026951 5995 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026954 5995 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026956 5995 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026959 5995 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026961 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026964 5995 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026966 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026969 5995 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026971 5995 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026974 5995 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026977 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026979 5995 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026982 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026984 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026986 5995 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026989 5995 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026991 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:10:50.027401 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026994 5995 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026997 5995 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.026999 5995 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027002 5995 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027004 5995 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027014 5995 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027017 5995 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027019 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027022 5995 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027027 5995 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027031 5995 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027034 5995 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027037 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027040 5995 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027044 5995 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027047 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027050 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027052 5995 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027055 5995 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:10:50.028134 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027057 5995 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027060 5995 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027063 5995 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027066 5995 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027069 5995 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027072 5995 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027074 5995 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027077 5995 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027080 5995 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027082 5995 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027085 5995 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027088 5995 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027091 5995 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027093 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027096 5995 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027099 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027101 5995 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027104 5995 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027107 5995 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:10:50.028604 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027110 5995 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027113 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027116 5995 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027118 5995 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027121 5995 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027123 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027126 5995 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027128 5995 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027131 5995 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027133 5995 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027136 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027139 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027142 5995 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027144 5995 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027147 5995 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027149 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027152 5995 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027155 5995 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027157 5995 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027160 5995 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:10:50.029043 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027163 5995 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027165 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027168 5995 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027170 5995 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027173 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027177 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027179 5995 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027182 5995 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027956 5995 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027964 5995 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027967 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027970 5995 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027974 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027977 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027980 5995 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027983 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027986 5995 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027989 5995 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027991 5995 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027994 5995 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:10:50.029523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027997 5995 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.027999 5995 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028002 5995 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028005 5995 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028008 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028010 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028013 5995 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028016 5995 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028020 5995 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028023 5995 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028026 5995 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028029 5995 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028032 5995 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028035 5995 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028037 5995 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028040 5995 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028042 5995 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028045 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028047 5995 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:10:50.029997 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028050 5995 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028052 5995 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028055 5995 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028057 5995 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028061 5995 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028063 5995 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028066 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028072 5995 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028075 5995 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028078 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028080 5995 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028082 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028085 5995 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028087 5995 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028090 5995 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028092 5995 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028096 5995 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028098 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028101 5995 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028103 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:10:50.030493 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028106 5995 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028109 5995 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028111 5995 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028113 5995 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028116 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028119 5995 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028121 5995 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028124 5995 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028127 5995 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028129 5995 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028132 5995 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028135 5995 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028138 5995 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028140 5995 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028143 5995 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028145 5995 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028149 5995 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028151 5995 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028154 5995 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028156 5995 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028159 5995 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:10:50.030968 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028162 5995 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028165 5995 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028167 5995 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028170 5995 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028173 5995 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028175 5995 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028178 5995 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028181 5995 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028183 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028186 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028190 5995 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028193 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028196 5995 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028198 5995 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028201 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028203 5995 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028205 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028208 5995 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028210 5995 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:10:50.031503 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028213 5995 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028215 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028217 5995 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028220 5995 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028222 5995 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028225 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028227 5995 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028230 5995 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028233 5995 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028236 5995 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028238 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028241 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028243 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028246 5995 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028249 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028769 5995 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028776 5995 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028779 5995 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028782 5995 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:10:50.031946 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028785 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028788 5995 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028790 5995 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028793 5995 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028796 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028798 5995 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028801 5995 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028803 5995 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028806 5995 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028808 5995 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028811 5995 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028814 5995 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028818 5995 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028821 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028825 5995 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028828 5995 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028830 5995 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028833 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028835 5995 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:10:50.032378 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028838 5995 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028841 5995 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028843 5995 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028846 5995 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028848 5995 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028851 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028854 5995 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028857 5995 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028860 5995 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028862 5995 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028865 5995 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028868 5995 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028871 5995 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028874 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028876 5995 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028878 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028881 5995 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028884 5995 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028886 5995 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028889 5995 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:10:50.033088 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028891 5995 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028894 5995 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028899 5995 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028902 5995 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028904 5995 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028907 5995 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028909 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028912 5995 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028915 5995 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028917 5995 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028920 5995 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028922 5995 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028925 5995 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028927 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028931 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028933 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028937 5995 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028940 5995 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028942 5995 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028944 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:10:50.033937 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028947 5995 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028950 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028953 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028955 5995 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028958 5995 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028960 5995 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028962 5995 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028965 5995 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028968 5995 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028970 5995 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028973 5995 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028976 5995 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028978 5995 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028981 5995 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028984 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028987 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028989 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028992 5995 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028994 5995 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:10:50.034828 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.028997 5995 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029000 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029002 5995 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029005 5995 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029008 5995 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029010 5995 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029013 5995 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029016 5995 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029018 5995 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029021 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029023 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029026 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029028 5995 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029031 5995 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029034 5995 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029036 5995 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029039 5995 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029041 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029044 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:10:50.035661 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029047 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029049 5995 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029052 5995 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029057 5995 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029060 5995 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029150 5995 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029161 5995 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029168 5995 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029174 5995 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029179 5995 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029182 5995 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029186 5995 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029190 5995 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029193 5995 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029196 5995 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029200 5995 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029203 5995 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029206 5995 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029209 5995 flags.go:64] FLAG: --cgroup-root="" Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029212 5995 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029216 5995 flags.go:64] FLAG: --client-ca-file="" Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029219 5995 flags.go:64] FLAG: --cloud-provider="external" Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029222 5995 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 01:10:50.036474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029226 5995 flags.go:64] FLAG: --cluster-domain="" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029229 5995 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029232 5995 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029235 5995 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029239 5995 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029243 5995 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029247 5995 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029250 5995 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029253 5995 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029256 5995 flags.go:64] FLAG: --contention-profiling="false" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029259 5995 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029262 5995 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029265 5995 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029268 5995 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029273 5995 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029276 5995 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029279 5995 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029285 5995 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029288 5995 flags.go:64] FLAG: --enable-server="true" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029291 5995 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029295 5995 flags.go:64] FLAG: --event-burst="100" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029298 5995 flags.go:64] FLAG: --event-qps="50" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029301 5995 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029305 5995 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 01:10:50.037557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029308 5995 flags.go:64] FLAG: --eviction-hard="" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029312 5995 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029315 5995 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029318 5995 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029321 5995 flags.go:64] FLAG: --eviction-soft="" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029325 5995 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029328 5995 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029331 5995 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029334 5995 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029337 5995 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029340 5995 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029343 5995 flags.go:64] FLAG: --feature-gates="" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029348 5995 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029351 5995 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029354 5995 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029358 5995 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029361 5995 flags.go:64] FLAG: --healthz-port="10248" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029364 5995 flags.go:64] FLAG: --help="false" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029367 5995 flags.go:64] FLAG: --hostname-override="ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029370 5995 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029373 5995 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029376 5995 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029379 5995 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 01:10:50.038618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029382 5995 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029385 5995 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029390 5995 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029392 5995 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029395 5995 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029400 5995 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029403 5995 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029405 5995 flags.go:64] FLAG: --kube-reserved="" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029408 5995 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029425 5995 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029429 5995 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029432 5995 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029435 5995 flags.go:64] FLAG: --lock-file="" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029438 5995 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029441 5995 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029444 5995 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029449 5995 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029452 5995 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029455 5995 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029458 5995 flags.go:64] FLAG: --logging-format="text" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029461 5995 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029464 5995 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029467 5995 flags.go:64] FLAG: --manifest-url="" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029470 5995 flags.go:64] FLAG: --manifest-url-header="" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029475 5995 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 01:10:50.039625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029478 5995 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029482 5995 flags.go:64] FLAG: --max-pods="110" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029485 5995 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029488 5995 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029491 5995 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029494 5995 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029497 5995 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029499 5995 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029502 5995 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029510 5995 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029514 5995 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029517 5995 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029521 5995 flags.go:64] FLAG: --pod-cidr="" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029524 5995 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029527 5995 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029530 5995 flags.go:64] FLAG: --pods-per-core="0" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029533 5995 flags.go:64] FLAG: --port="10250" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029536 5995 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029539 5995 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-09334c893b020cf8c" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029542 5995 flags.go:64] FLAG: --qos-reserved="" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029545 5995 flags.go:64] FLAG: --read-only-port="10255" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029548 5995 flags.go:64] FLAG: --register-node="true" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029551 5995 flags.go:64] FLAG: --register-with-taints="" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029555 5995 flags.go:64] FLAG: --registry-burst="10" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029558 5995 flags.go:64] FLAG: --registry-qps="5" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029561 5995 flags.go:64] FLAG: --reserved-cpus="" Nov 04 01:10:50.040632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029564 5995 flags.go:64] FLAG: --reserved-memory="" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029568 5995 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029571 5995 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029574 5995 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029576 5995 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029579 5995 flags.go:64] FLAG: --runonce="false" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029582 5995 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029586 5995 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029589 5995 flags.go:64] FLAG: --seccomp-default="false" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029591 5995 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029594 5995 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029597 5995 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029600 5995 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029603 5995 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029606 5995 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029610 5995 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029613 5995 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029616 5995 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029620 5995 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029623 5995 flags.go:64] FLAG: --system-cgroups="" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029626 5995 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029631 5995 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029634 5995 flags.go:64] FLAG: --tls-cert-file="" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029637 5995 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029640 5995 flags.go:64] FLAG: --tls-min-version="" Nov 04 01:10:50.041462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029643 5995 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029646 5995 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029649 5995 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029652 5995 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029654 5995 flags.go:64] FLAG: --v="2" Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029663 5995 flags.go:64] FLAG: --version="false" Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029667 5995 flags.go:64] FLAG: --vmodule="" Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029671 5995 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.029674 5995 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029803 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029806 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029809 5995 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029812 5995 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029815 5995 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029818 5995 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029821 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029824 5995 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029826 5995 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029829 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029831 5995 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029834 5995 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:10:50.042045 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029836 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029840 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029843 5995 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029846 5995 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029849 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029852 5995 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029855 5995 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029857 5995 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029860 5995 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029863 5995 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029866 5995 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029868 5995 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029871 5995 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029873 5995 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029876 5995 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029878 5995 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029881 5995 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029883 5995 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029886 5995 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029889 5995 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:10:50.042634 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029891 5995 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029893 5995 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029896 5995 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029899 5995 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029901 5995 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029904 5995 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029907 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029909 5995 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029912 5995 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029914 5995 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029917 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029919 5995 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029922 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029925 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029928 5995 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029930 5995 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029935 5995 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029938 5995 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029940 5995 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029943 5995 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:10:50.043160 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029945 5995 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029948 5995 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029950 5995 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029953 5995 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029955 5995 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029958 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029961 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029963 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029966 5995 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029968 5995 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029971 5995 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029973 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029976 5995 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029978 5995 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029981 5995 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029983 5995 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029986 5995 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029988 5995 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029991 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:10:50.043753 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029994 5995 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029997 5995 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.029999 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030002 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030005 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030008 5995 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030012 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030016 5995 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030019 5995 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030023 5995 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030026 5995 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030029 5995 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030031 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030033 5995 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030036 5995 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030038 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030041 5995 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030043 5995 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030047 5995 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:10:50.044317 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030051 5995 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:10:50.044846 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030054 5995 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:10:50.044846 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030057 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:10:50.044846 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030059 5995 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:10:50.044846 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030062 5995 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:10:50.044846 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030064 5995 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:10:50.044846 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030067 5995 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:10:50.044846 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030069 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:10:50.044846 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030072 5995 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:10:50.044846 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030075 5995 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:10:50.044846 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030077 5995 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:10:50.044846 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030080 5995 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:10:50.044846 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030083 5995 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:10:50.044846 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030085 5995 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:10:50.044846 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030088 5995 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:10:50.044846 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030090 5995 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:10:50.044846 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.030116 5995 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030150 5995 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030155 5995 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030158 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030162 5995 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030165 5995 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030170 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030172 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030175 5995 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030178 5995 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030180 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030183 5995 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030185 5995 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030188 5995 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030198 5995 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030202 5995 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030205 5995 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030208 5995 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030210 5995 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030213 5995 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:10:50.045273 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030215 5995 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030218 5995 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030220 5995 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030223 5995 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030226 5995 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030246 5995 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030250 5995 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030252 5995 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030255 5995 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030257 5995 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030260 5995 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030264 5995 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030266 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030269 5995 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030272 5995 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030274 5995 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030279 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030281 5995 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030285 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030288 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:10:50.046020 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030290 5995 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030293 5995 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030296 5995 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030299 5995 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030301 5995 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030304 5995 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030307 5995 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030309 5995 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030312 5995 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030315 5995 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030317 5995 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030320 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030323 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030326 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030328 5995 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030331 5995 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030334 5995 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030336 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030339 5995 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:10:50.046712 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030342 5995 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030344 5995 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030347 5995 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030349 5995 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030352 5995 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030355 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030358 5995 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030362 5995 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030365 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030369 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030372 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030376 5995 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030378 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030382 5995 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030387 5995 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030391 5995 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030394 5995 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030398 5995 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030401 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:10:50.047245 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030404 5995 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030407 5995 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030409 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030425 5995 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030429 5995 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030435 5995 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030441 5995 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030446 5995 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030452 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030456 5995 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030460 5995 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030463 5995 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030467 5995 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030469 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030472 5995 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030475 5995 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030477 5995 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030480 5995 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030483 5995 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:10:50.047742 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030486 5995 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030489 5995 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030492 5995 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030497 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030500 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030504 5995 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030506 5995 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030509 5995 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030512 5995 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.030515 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.035285 5995 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.035311 5995 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.035335 5995 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.035343 5995 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035458 5995 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035466 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035471 5995 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035476 5995 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035480 5995 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:10:50.048255 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035485 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035489 5995 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035493 5995 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035498 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035503 5995 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035507 5995 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035511 5995 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035515 5995 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035520 5995 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035524 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035528 5995 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035532 5995 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035537 5995 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035542 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035546 5995 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035551 5995 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035555 5995 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035560 5995 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035564 5995 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035568 5995 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:10:50.048812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035573 5995 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035577 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035581 5995 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035586 5995 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035600 5995 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035605 5995 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035609 5995 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035613 5995 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035617 5995 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035622 5995 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035626 5995 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035633 5995 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035640 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035645 5995 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035651 5995 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035655 5995 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035659 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035663 5995 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035667 5995 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:10:50.049285 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035671 5995 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035675 5995 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035679 5995 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035683 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035687 5995 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035692 5995 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035696 5995 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035700 5995 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035705 5995 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035711 5995 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035717 5995 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035723 5995 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035727 5995 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035732 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035736 5995 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035741 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035746 5995 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035750 5995 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035763 5995 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:10:50.049820 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035767 5995 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035772 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035776 5995 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035780 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035785 5995 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035789 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035794 5995 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035798 5995 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035803 5995 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035807 5995 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035811 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035815 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035819 5995 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035823 5995 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035828 5995 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035832 5995 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035836 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035841 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035845 5995 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:10:50.050301 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035849 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035854 5995 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035858 5995 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035862 5995 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035866 5995 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035870 5995 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035874 5995 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035879 5995 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035883 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035888 5995 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035892 5995 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035897 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035901 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035913 5995 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035918 5995 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035922 5995 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035926 5995 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035930 5995 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035934 5995 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035938 5995 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:10:50.050792 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035942 5995 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:10:50.051316 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035946 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:10:50.051316 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035951 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:10:50.051316 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.035955 5995 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:10:50.051316 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.035995 5995 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 01:10:50.051316 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036289 5995 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:10:50.051316 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036297 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:10:50.051316 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036302 5995 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:10:50.051316 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036307 5995 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:10:50.051316 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036312 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:10:50.051316 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036316 5995 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:10:50.051316 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036320 5995 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:10:50.051316 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036324 5995 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:10:50.051316 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036328 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:10:50.051316 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036333 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:10:50.051316 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036337 5995 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:10:50.051316 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036341 5995 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036346 5995 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036350 5995 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036354 5995 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036358 5995 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036363 5995 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036367 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036371 5995 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036376 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036380 5995 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036394 5995 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036398 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036402 5995 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036406 5995 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036433 5995 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036439 5995 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036444 5995 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036448 5995 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036452 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036457 5995 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:10:50.051731 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036461 5995 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036465 5995 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036469 5995 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036473 5995 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036478 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036482 5995 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036486 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036490 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036494 5995 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036498 5995 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036502 5995 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036506 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036511 5995 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036515 5995 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036520 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036524 5995 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036528 5995 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036533 5995 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036537 5995 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:10:50.052201 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036541 5995 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036545 5995 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036550 5995 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036554 5995 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036566 5995 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036571 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036576 5995 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036580 5995 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036585 5995 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036589 5995 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036593 5995 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036598 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036603 5995 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036607 5995 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036611 5995 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036615 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036620 5995 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036624 5995 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036628 5995 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:10:50.052773 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036640 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036645 5995 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036649 5995 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036654 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036658 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036662 5995 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036666 5995 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036670 5995 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036674 5995 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036678 5995 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036682 5995 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036687 5995 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036691 5995 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036695 5995 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036699 5995 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036703 5995 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036707 5995 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036711 5995 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036723 5995 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036728 5995 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:10:50.053211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036732 5995 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:10:50.053732 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036736 5995 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:10:50.053732 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036740 5995 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:10:50.053732 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036747 5995 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:10:50.053732 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036753 5995 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:10:50.053732 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036758 5995 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:10:50.053732 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036762 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:10:50.053732 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036766 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:10:50.053732 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036771 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:10:50.053732 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036777 5995 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:10:50.053732 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036782 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:10:50.053732 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036787 5995 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:10:50.053732 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036791 5995 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:10:50.053732 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036795 5995 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:10:50.053732 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036799 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:10:50.053732 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036803 5995 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:10:50.053732 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036807 5995 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:10:50.054281 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.036848 5995 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 01:10:50.054281 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036906 5995 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 01:10:50.054281 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036913 5995 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 01:10:50.054281 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036917 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 01:10:50.054281 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036921 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 01:10:50.054281 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036925 5995 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 01:10:50.054281 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036929 5995 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 01:10:50.054281 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036934 5995 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 01:10:50.054281 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036938 5995 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 01:10:50.054281 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036942 5995 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 01:10:50.054281 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036946 5995 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 01:10:50.054281 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036951 5995 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 01:10:50.054281 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036955 5995 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 01:10:50.054281 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036959 5995 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 01:10:50.054281 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036972 5995 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 01:10:50.054281 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036976 5995 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036980 5995 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036985 5995 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036989 5995 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036993 5995 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.036997 5995 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037001 5995 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037005 5995 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037012 5995 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037019 5995 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037024 5995 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037029 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037034 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037038 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037044 5995 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037049 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037055 5995 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037059 5995 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037063 5995 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 01:10:50.054860 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037067 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037071 5995 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037075 5995 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037079 5995 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037084 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037088 5995 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037092 5995 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037097 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037101 5995 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037105 5995 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037109 5995 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037113 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037117 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037129 5995 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037138 5995 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037143 5995 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037147 5995 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037151 5995 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037155 5995 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037160 5995 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 01:10:50.055541 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037164 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037168 5995 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037173 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037177 5995 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037181 5995 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037185 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037190 5995 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037194 5995 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037198 5995 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037202 5995 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037207 5995 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037211 5995 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037216 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037220 5995 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037224 5995 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037229 5995 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037233 5995 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037238 5995 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037242 5995 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037246 5995 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 01:10:50.056229 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037250 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037254 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037259 5995 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037263 5995 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037268 5995 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037272 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037287 5995 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037292 5995 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037296 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037300 5995 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037304 5995 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037308 5995 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037312 5995 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037317 5995 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037321 5995 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037325 5995 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037330 5995 feature_gate.go:386] unrecognized feature gate: Example Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037334 5995 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037338 5995 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 01:10:50.056922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037343 5995 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 01:10:50.057610 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037347 5995 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 01:10:50.057610 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037353 5995 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 01:10:50.057610 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037357 5995 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 01:10:50.057610 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037361 5995 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 01:10:50.057610 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037365 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 01:10:50.057610 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037370 5995 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 01:10:50.057610 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037374 5995 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 01:10:50.057610 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037378 5995 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 01:10:50.057610 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037385 5995 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 01:10:50.057610 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037389 5995 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 01:10:50.057610 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037393 5995 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 01:10:50.057610 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037397 5995 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 01:10:50.057610 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:50.037401 5995 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 01:10:50.057610 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.037500 5995 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 01:10:50.057610 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.039141 5995 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 04 01:10:50.057610 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.039207 5995 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 04 01:10:50.057610 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.039808 5995 server.go:1014] "Starting client certificate rotation" Nov 04 01:10:50.057610 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.040165 5995 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 01:10:50.058269 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.040396 5995 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 01:04:36 +0000 UTC" deadline="2028-08-04 19:55:11.266496046 +0000 UTC" Nov 04 01:10:50.058269 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.040533 5995 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15354h44m21.22596694s" Nov 04 01:10:50.058269 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.041305 5995 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 01:10:50.058269 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.042391 5995 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 01:10:50.058269 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.043850 5995 log.go:25] "Validated CRI v1 runtime API" Nov 04 01:10:50.058269 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.045842 5995 log.go:25] "Validated CRI v1 image API" Nov 04 01:10:50.058269 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.046630 5995 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 01:10:50.058269 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.049046 5995 fs.go:135] Filesystem UUIDs: map[735397ad-cda3-45de-883c-dbbcfb80bb89:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 972bd2db-728e-44e9-9585-9de78c2caa26:/dev/nvme0n1p4] Nov 04 01:10:50.058640 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.049067 5995 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0750204d2e8b59273e5d36bb112a2603bbe424206ef65dd634bb6a7a3efa927d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0750204d2e8b59273e5d36bb112a2603bbe424206ef65dd634bb6a7a3efa927d/userdata/shm major:0 minor:369 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12b8e2e7d0b1cfe4c92e5f13d3fe78740de4021fc7f21746a5bee63bba5a4ec4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12b8e2e7d0b1cfe4c92e5f13d3fe78740de4021fc7f21746a5bee63bba5a4ec4/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/13d5686dc21a2275c049fb9f51447a8f8a3652ae89ee45e5e95bc5efead89a01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/13d5686dc21a2275c049fb9f51447a8f8a3652ae89ee45e5e95bc5efead89a01/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18cf5ce0a486bf5b432b3279552d68791ee83754ac6cc2a8bf87576938774556/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18cf5ce0a486bf5b432b3279552d68791ee83754ac6cc2a8bf87576938774556/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f80bab462e52ddc7607885b1de473b4caed00db64ad6f17487d337795134196/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f80bab462e52ddc7607885b1de473b4caed00db64ad6f17487d337795134196/userdata/shm major:0 minor:379 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/251e21ccd12219fcf62ea2f806f78dc2c3356ebc03f66d402fbcd980dd360e8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/251e21ccd12219fcf62ea2f806f78dc2c3356ebc03f66d402fbcd980dd360e8a/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29855f357f0133564c0d8111980f94532023567cc7f628e722b906f68c1c6a2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29855f357f0133564c0d8111980f94532023567cc7f628e722b906f68c1c6a2c/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30e0df48fcd09cdce51e26071a7daec8132477b4ae20bb7af8f568959ad13341/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30e0df48fcd09cdce51e26071a7daec8132477b4ae20bb7af8f568959ad13341/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36d411089d38ab9155869fc5fc027a051a82de1059856529daf993a1c77e0a67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36d411089d38ab9155869fc5fc027a051a82de1059856529daf993a1c77e0a67/userdata/shm major:0 minor:256 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/44d7b0a2497bef282b17ba2e0e4b41b6cca26c8d3a27b5de2f18c3746254701d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/44d7b0a2497bef282b17ba2e0e4b41b6cca26c8d3a27b5de2f18c3746254701d/userdata/shm major:0 minor:512 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a82c2a760402ddb4b1ed8bacf6330c70ac99acd53cf7e314cc7d25d2280272e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a82c2a760402ddb4b1ed8bacf6330c70ac99acd53cf7e314cc7d25d2280272e/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f47617b789c3b3cb03076d6552d01b677d953d5f1021e70bb844f965e7f17cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f47617b789c3b3cb03076d6552d01b677d953d5f1021e70bb844f965e7f17cf/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57632676aacede02ef6f25328a842b6cdddff2f9f7290826b4767f5643f043f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57632676aacede02ef6f25328a842b6cdddff2f9f7290826b4767f5643f043f0/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5bc62a7e61ebb236bb4631d93bab8331783661fd6a433c1d3f2744be9b012835/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5bc62a7e61ebb236bb4631d93bab8331783661fd6a433c1d3f2744be9b012835/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ecbf13bab42e195b5371678969094ac07ed75be0035a7afed257df0aadbe0af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ecbf13bab42e195b5371678969094ac07ed75be0035a7afed257df0aadbe0af/userdata/shm major:0 minor:457 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65ee99eef508daea64d5d7e15539739e09ad12a1e442f4067af4dc52aabf6ba0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65ee99eef508daea64d5d7e15539739e09ad12a1e442f4067af4dc52aabf6ba0/userdata/shm major:0 minor:510 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69a81c3f70096360826bcc68061e4a63af85de5c099c0524fa9ba022f6671d82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69a81c3f70096360826bcc68061e4a63af85de5c099c0524fa9ba022f6671d82/userdata/shm major:0 minor:481 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c93d8fe3521e9d82ee6651bc3b234e56a2e47a042a2b191d87d595990d2def2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c93d8fe3521e9d82ee6651bc3b234e56a2e47a042a2b191d87d595990d2def2/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/738361f938aa7c6d4c807186a4dced11bcc6dbea29c8580aeff49fc3f28ff4f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/738361f938aa7c6d4c807186a4dced11bcc6dbea29c8580aeff49fc3f28ff4f2/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76b3d30e17580e47f2e408a0debe8067481a8f8432dc342f8f86a0b6cc06020d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76b3d30e17580e47f2e408a0debe8067481a8f8432dc342f8f86a0b6cc06020d/userdata/shm major:0 minor:376 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/83ffd8448ded9b9ec3ef6209f1bf5d4b96993fb173ab544e4321f7611f36e6ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/83ffd8448ded9b9ec3ef6209f1bf5d4b96993fb173ab544e4321f7611f36e6ce/userdata/shm major:0 minor:320 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86a37a601387a008f2c0ddd8f57e96ab06eb284bd0203263892fec8bfbd4aec4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86a37a601387a008f2c0ddd8f57e96ab06eb284bd0203263892fec8bfbd4aec4/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/875c7ad36433ca2510d1f9bc776d1abf6a9ee2bef42bae468edd1159927d291a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/875c7ad36433ca2510d1f9bc776d1abf6a9ee2bef42bae468edd1159927d291a/userdata/shm major:0 minor:479 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c35ab28fb00962801a8fe34285bf6942fc29ad7360a9e7604f957437f7d950f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c35ab28fb00962801a8fe34285bf6942fc29ad7360a9e7604f957437f7d950f/userdata/shm major:0 minor:464 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9544ea47f6ea8e8786d664c14ef2178951957028bd8ff037e155ae0b8a532e82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9544ea47f6ea8e8786d664c14ef2178951957028bd8ff037e155ae0b8a532e82/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97eefcf4cb3d26c1db3a3ea4102ab8d0e9226c78b7451d734136ca782bf547f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97eefcf4cb3d26c1db3a3ea4102ab8d0e9226c78b7451d734136ca782bf547f8/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98172af2c8dfd62512835ca75a3182e74da317adbe908f18a05b0e0c39af5bfa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98172af2c8dfd62512835ca75a3182e74da317adbe908f18a05b0e0c39af5bfa/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98da5efe82e0b952f83c1d1541ad31ac5ab87b519f7a25be001d0ab695443267/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98da5efe82e0b952f83c1d1541ad31ac5ab87b519f7a25be001d0ab695443267/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a02e524282e41a87c6ab6ad18b28a081e82927589533c29c68df9f4ea7a8a94c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a02e524282e41a87c6ab6ad18b28a081e82927589533c29c68df9f4ea7a8a94c/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a889499cabdba0d2fd9a2ef3f5313b44a484207a29a7b6f6f795fcf287ea34be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a889499cabdba0d2fd9a2ef3f5313b44a484207a29a7b6f6f795fcf287ea34be/userdata/shm major:0 minor:454 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8df2f8562b1d8be2e8110165044fe414fe196d1fa649d8d0218c9d4dce16a11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8df2f8562b1d8be2e8110165044fe414fe196d1fa649d8d0218c9d4dce16a11/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/acf4fc0ba80c720332e88d492e223ff678c13b3e7604db81bd64148c02b40836/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/acf4fc0ba80c720332e88d492e223ff678c13b3e7604db81bd64148c02b40836/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7f25d5d02c71d60f4299230aed43c4c7113dfb0a34777bf62d08ea66c862290/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7f25d5d02c71d60f4299230aed43c4c7113dfb0a34777bf62d08ea66c862290/userdata/shm major:0 minor:477 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6c384c5ee28e82f224c188c97a0325dd539c534432b42c0bf543d8adaaa8a9e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6c384c5ee28e82f224c188c97a0325dd539c534432b42c0bf543d8adaaa8a9e/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfb374093c7ff29f6badcdd2465ae551ee208b14450e298c52bc5a9384416355/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfb374093c7ff29f6badcdd2465ae551ee208b14450e298c52bc5a9384416355/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2dd755b06f3e13f4f7881c0031f294ae2d40a05f702199edab0368f01ee59c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2dd755b06f3e13f4f7881c0031f294ae2d40a05f702199edab0368f01ee59c6/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6fbf21c0ddf595ca72a79c5e68ff7f45cd411422580b519df444546ce79f4f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6fbf21c0ddf595ca72a79c5e68ff7f45cd411422580b519df444546ce79f4f6/userdata/shm major:0 minor:433 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03f3058b-0663-48a7-ab67-a41bfe5a2c57/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/03f3058b-0663-48a7-ab67-a41bfe5a2c57/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03f3058b-0663-48a7-ab67-a41bfe5a2c57/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/03f3058b-0663-48a7-ab67-a41bfe5a2c57/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03f3058b-0663-48a7-ab67-a41bfe5a2c57/volumes/kubernetes.io~projected/kube-api-access-rlf5p:{mountpoint:/var/lib/kubelet/pods/03f3058b-0663-48a7-ab67-a41bfe5a2c57/volumes/kubernetes.io~projected/kube-api-access-rlf5p major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f50bc9b-91da-4d92-8553-c9f9476665e4/volumes/kubernetes.io~projected/kube-api-access-zhhtl:{mountpoint:/var/lib/kubelet/pods/0f50bc9b-91da-4d92-8553-c9f9476665e4/volumes/kubernetes.io~projected/kube-api-access-zhhtl major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ae06262-6e59-4775-bedc-5305ace1fae6/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3ae06262-6e59-4775-bedc-5305ace1fae6/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:475 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42f02121-70f2-43d1-be9d-2a81425ec97d/volumes/kubernetes.io~projected/kube-api-access-sxlfx:{mountpoint:/var/lib/kubelet/pods/42f02121-70f2-43d1-be9d-2a81425ec97d/volumes/kubernetes.io~projected/kube-api-access-sxlfx major:0 minor:476 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b8947a9-b2f0-4de2-a1b8-58342b6b3240/volumes/kubernetes.io~secret/networking-console-plugin-cert:{mountpoint:/var/lib/kubelet/pods/5b8947a9-b2f0-4de2-a1b8-58342b6b3240/volumes/kubernetes.io~secret/networking-console-plugin-cert major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cecaa47-44ef-440e-80eb-45230fe3edc0/volumes/kubernetes.io~projected/kube-api-access-d94r5:{mountpoint:/var/lib/kubelet/pods/5cecaa47-44ef-440e-80eb-45230fe3edc0/volumes/kubernetes.io~projected/kube-api-access-d94r5 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cecaa47-44ef-440e-80eb-45230fe3edc0/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5cecaa47-44ef-440e-80eb-45230fe3edc0/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d1de04c-5111-415b-bdcc-20dfa9d0e6f1/volumes/kubernetes.io~projected/kube-api-access-j8rgj:{mountpoint:/var/lib/kubelet/pods/5d1de04c-5111-415b-bdcc-20dfa9d0e6f1/volumes/kubernetes.io~projected/kube-api-access-j8rgj major:0 minor:456 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6250a398-c824-4c6e-9083-c7b0df543700/volumes/kubernetes.io~projected/kube-api-access-p78dk:{mountpoint:/var/lib/kubelet/pods/6250a398-c824-4c6e-9083-c7b0df543700/volumes/kubernetes.io~projected/kube-api-access-p78dk major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/747ed7ed-d05b-434a-8360-acc2292c2257/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/747ed7ed-d05b-434a-8360-acc2292c2257/volumes/kubernetes.io~secret/tls-certificates major:0 minor:450 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7573b3c8-fd6e-41c5-9de4-9c441f2ce026/volumes/kubernetes.io~projected/kube-api-access-dp5d9:{mountpoint:/var/lib/kubelet/pods/7573b3c8-fd6e-41c5-9de4-9c441f2ce026/volumes/kubernetes.io~projected/kube-api-access-dp5d9 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7573b3c8-fd6e-41c5-9de4-9c441f2ce026/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/7573b3c8-fd6e-41c5-9de4-9c441f2ce026/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75c5b017-8f43-41dc-b79b-0353de290289/volumes/kubernetes.io~projected/kube-api-access-dm4k6:{mountpoint:/var/lib/kubelet/pods/75c5b017-8f43-41dc-b79b-0353de290289/volumes/kubernetes.io~projected/kube-api-access-dm4k6 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75c5b017-8f43-41dc-b79b-0353de290289/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls:{mountpoint:/var/lib/kubelet/pods/75c5b017-8f43-41dc-b79b-0353de290289/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:463 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe/volumes/kubernetes.io~projected/kube-api-access-xlvzh:{mountpoint:/var/lib/kubelet/pods/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe/volumes/kubernetes.io~projected/kube-api-access-xlvzh major:0 minor:462 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe/volumes/kubernetes.io~projected/registry-tls major:0 minor:459 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:461 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:460 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82966365-6bec-467d-9279-12ac45dd01e2/volumes/kubernetes.io~projected/kube-api-access-fpmgk:{mountpoint:/var/lib/kubelet/pods/82966365-6bec-467d-9279-12ac45dd01e2/volumes/kubernetes.io~projected/kube-api-access-fpmgk major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89bd09a4-6d44-49ae-9bfa-837e2e5bf915/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/89bd09a4-6d44-49ae-9bfa-837e2e5bf915/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89bd09a4-6d44-49ae-9bfa-837e2e5bf915/volumes/kubernetes.io~projected/kube-api-access-bk7ck:{mountpoint:/var/lib/kubelet/pods/89bd09a4-6d44-49ae-9bfa-837e2e5bf915/volumes/kubernetes.io~projected/kube-api-access-bk7ck major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89bd09a4-6d44-49ae-9bfa-837e2e5bf915/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/89bd09a4-6d44-49ae-9bfa-837e2e5bf915/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5/volumes/kubernetes.io~projected/kube-api-access-ltmvj:{mountpoint:/var/lib/kubelet/pods/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5/volumes/kubernetes.io~projected/kube-api-access-ltmvj major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ddedc2c-735e-41e5-a5a9-3386ed890ee1/volumes/kubernetes.io~projected/kube-api-access-kxsvl:{mountpoint:/var/lib/kubelet/pods/8ddedc2c-735e-41e5-a5a9-3386ed890ee1/volumes/kubernetes.io~projected/kube-api-access-kxsvl major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a06edce6-baf3-40bc-9d0e-a6e71c393165/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a06edce6-baf3-40bc-9d0e-a6e71c393165/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a23c835a-aacb-4f92-a2cd-799f5e870b85/volumes/kubernetes.io~projected/kube-api-access-kpfww:{mountpoint:/var/lib/kubelet/pods/a23c835a-aacb-4f92-a2cd-799f5e870b85/volumes/kubernetes.io~projected/kube-api-access-kpfww major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7991c77-7106-4c73-8812-ff6d4c06b666/volumes/kubernetes.io~projected/kube-api-access-b7mxs:{mountpoint:/var/lib/kubelet/pods/a7991c77-7106-4c73-8812-ff6d4c06b666/volumes/kubernetes.io~projected/kube-api-access-b7mxs major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8b2b155-18ef-40a1-b87b-a6e44e130d65/volumes/kubernetes.io~projected/kube-api-access-9dxcj:{mountpoint:/var/lib/kubelet/pods/a8b2b155-18ef-40a1-b87b-a6e44e130d65/volumes/kubernetes.io~projected/kube-api-access-9dxcj major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae64edd1-43dc-406d-bcad-f7f34bb1f991/volumes/kubernetes.io~projected/kube-api-access-v22h7:{mountpoint:/var/lib/kubelet/pods/ae64edd1-43dc-406d-bcad-f7f34bb1f991/volumes/kubernetes.io~projected/kube-api-access-v22h7 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae64edd1-43dc-406d-bcad-f7f34bb1f991/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ae64edd1-43dc-406d-bcad-f7f34bb1f991/volumes/kubernetes.io~secret/serving-cert major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b44a2a16-b00a-4113-b0ab-5cdcb955b325/volumes/kubernetes.io~projected/kube-api-access-2pbbl:{mountpoint:/var/lib/kubelet/pods/b44a2a16-b00a-4113-b0ab-5cdcb955b325/volumes/kubernetes.io~projected/kube-api-access-2pbbl major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b44a2a16-b00a-4113-b0ab-5cdcb955b325/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b44a2a16-b00a-4113-b0ab-5cdcb955b325/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2aa236a-4ff4-4b39-90ff-c782721444ee/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/c2aa236a-4ff4-4b39-90ff-c782721444ee/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2aa236a-4ff4-4b39-90ff-c782721444ee/volumes/kubernetes.io~projected/kube-api-access-cx7dg:{mountpoint:/var/lib/kubelet/pods/c2aa236a-4ff4-4b39-90ff-c782721444ee/volumes/kubernetes.io~projected/kube-api-access-cx7dg major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2aa236a-4ff4-4b39-90ff-c782721444ee/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/c2aa236a-4ff4-4b39-90ff-c782721444ee/volumes/kubernetes.io~projected/registry-tls major:0 minor:503 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2aa236a-4ff4-4b39-90ff-c782721444ee/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/c2aa236a-4ff4-4b39-90ff-c782721444ee/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2aa236a-4ff4-4b39-90ff-c782721444ee/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/c2aa236a-4ff4-4b39-90ff-c782721444ee/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3d08d00-e371-4793-955e-c5b51eef7879/volumes/kubernetes.io~projected/kube-api-access-v8xv5:{mountpoint:/var/lib/kubelet/pods/c3d08d00-e371-4793-955e-c5b51eef7879/volumes/kubernetes.io~projected/kube-api-access-v8xv5 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf/volumes/kubernetes.io~projected/kube-api-access-chq5x:{mountpoint:/var/lib/kubelet/pods/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf/volumes/kubernetes.io~projected/kube-api-access-chq5x major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf/volumes/kubernetes.io~secret/signing-key major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5b0a3f1-5189-4572-9765-20eb3960fbae/volumes/kubernetes.io~projected/kube-api-access-245fq:{mountpoint:/var/lib/kubelet/pods/c5b0a3f1-5189-4572-9765-20eb3960fbae/volumes/kubernetes.io~projected/kube-api-access-245fq major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5dd40be-e515-4fe8-be55-b93f5b97973e/volumes/kubernetes.io~projected/kube-api-access-2tgzg:{mountpoint:/var/lib/kubelet/pods/c5dd40be-e515-4fe8-be55-b93f5b97973e/volumes/kubernetes.io~projected/kube-api-access-2tgzg major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5dd40be-e515-4fe8-be55-b93f5b97973e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/c5dd40be-e515-4fe8-be55-b93f5b97973e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c60eb5db-ec53-40bb-9a72-c712cec7c32c/volumes/kubernetes.io~projected/kube-api-access-4b5fm:{mountpoint:/var/lib/kubelet/pods/c60eb5db-ec53-40bb-9a72-c712cec7c32c/volumes/kubernetes.io~projected/kube-api-access-4b5fm major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca77e439-49cb-4ef0-9951-9564ab28ed3e/volumes/kubernetes.io~projected/kube-api-access-lk265:{mountpoint:/var/lib/kubelet/pods/ca77e439-49cb-4ef0-9951-9564ab28ed3e/volumes/kubernetes.io~projected/kube-api-access-lk265 major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca77e439-49cb-4ef0-9951-9564ab28ed3e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/ca77e439-49cb-4ef0-9951-9564ab28ed3e/volumes/kubernetes.io~secret/cert major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd605a92-f88c-40ea-bde7-53c2da66a576/volumes/kubernetes.io~projected/kube-api-access-9tf5p:{mountpoint:/var/lib/kubelet/pods/cd605a92-f88c-40ea-bde7-53c2da66a576/volumes/kubernetes.io~projected/kube-api-access-9tf5p major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29/volumes/kubernetes.io~projected/kube-api-access-827hl:{mountpoint:/var/lib/kubelet/pods/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29/volumes/kubernetes.io~projected/kube-api-access-827hl major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e96cde4e-0238-48e6-b343-a0592f6e159e/volumes/kubernetes.io~projected/kube-api-access-ks2wj:{mountpoint:/var/lib/kubelet/pods/e96cde4e-0238-48e6-b343-a0592f6e159e/volumes/kubernetes.io~projected/kube-api-access-ks2wj major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e96cde4e-0238-48e6-b343-a0592f6e159e/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/e96cde4e-0238-48e6-b343-a0592f6e159e/volumes/kubernetes.io~secret/metrics-tls major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7aa61c-2789-49ed-989d-fa77fed0ff0f/volumes/kubernetes.io~projected/kube-api-access-gsq8f:{mountpoint:/var/lib/kubelet/pods/ef7aa61c-2789-49ed-989d-fa77fed0ff0f/volumes/kubernetes.io~projected/kube-api-access-gsq8f major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef7aa61c-2789-49ed-989d-fa77fed0ff0f/volumes/kubernetes.io~secret/samples-operator-tls:{mountpoint:/var/lib/kubelet/pods/ef7aa61c-2789-49ed-989d-fa77fed0ff0f/volumes/kubernetes.io~secret/samples-operator-tls major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2df8dc5-6403-4827-b6d6-9c209c595e05/volumes/kubernetes.io~projected/kube-api-access-jl9gz:{mountpoint:/var/lib/kubelet/pods/f2df8dc5-6403-4827-b6d6-9c209c595e05/volumes/kubernetes.io~projected/kube-api-access-jl9gz major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2df8dc5-6403-4827-b6d6-9c209c595e05/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/f2df8dc5-6403-4827-b6d6-9c209c595e05/volumes/kubernetes.io~secret/default-certificate major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2df8dc5-6403-4827-b6d6-9c209c595e05/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/f2df8dc5-6403-4827-b6d6-9c209c595e05/volumes/kubernetes.io~secret/metrics-certs major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2df8dc5-6403-4827-b6d6-9c209c595e05/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/f2df8dc5-6403-4827-b6d6-9c209c595e05/volumes/kubernetes.io~secret/stats-auth major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8/volumes/kubernetes.io~projected/kube-api-access-pm8jv:{mountpoint:/var/lib/kubelet/pods/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8/volumes/kubernetes.io~projected/kube-api-access-pm8jv major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8/volumes/kubernetes.io~secret/metrics-certs major:0 minor:470 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb55ed6-b225-4db5-a7fb-66a10868befc/volumes/kubernetes.io~projected/kube-api-access-slglc:{mountpoint:/var/lib/kubelet/pods/ffb55ed6-b225-4db5-a7fb-66a10868befc/volumes/kubernetes.io~projected/kube-api-access-slglc major:0 minor:64 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/bf7993aa8a378be622fe651d383c2250b3abe203038634953b0f48c8171528cf/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/0b23d19743f867ab720570a3c157601a4d2455dd72e0762925bdb348a8883971/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f1d3d27897fb2876d21374361aeadb168c1159903a5d774d09f0ed3d4c3de325/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/62cbd06dbe7714406b2aebb7478cf3e0fdeccf4a02fd16a6cea469f21f2ddf7a/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/fe4ca75d25631eb3b5289863475f4b94eae4cc0374d17d4ae92a82067f28eae1/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/bfb85dd979157a27aa7502bf8c17e376add15557c248a36889f90d715330d5cf/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/58aa697eab5ca4809797ddf202d01cf42241a094ddc7d8199b7008510c00d453/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a8145f84ce9812337c163033fb822407dc1215e2f4588b54a96b7b2bc3468668/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/8329615f78a7ad87982cad99eb7f59c382ff1089eca60b9047bcb0b9b2df7f3b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/fbe0bb228abb3d23989e66c6b37effec4c4aa377f8cfb590a76c5963d01ecb62/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/ba63e562487ce840513fcb7ae95ef32b82a8a9e66782fa89e170c19847e939a0/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/aa7aaceb0a2c7776bcdb594727c82a300795999ea2f399a96b11a88def423441/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/cf321b7bad6c6eeae461539daf0e89d38d7e984c1141c4f2192e422deda09c78/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1e9810fc9dff6f6697178e0da43634e99d14eaa526dd41c71ffa2f728ad49285/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/26e078234cdd99d4548c91bf5c20a7f5b974b2dd0b21661fefef7900aa2d740d/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/17d22a0b9c57558959c662a354e86253fd6553cdc3527fd916a9f2c2f4edd877/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b43aa071111dbea8ed85b2d0411644e61c801cc2f2e7d9de93a29691836357b9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e12b136df68ba5af154979ac4ba5413e4aa1a206338abc4168916ac280f4c54b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/42e23f21888d0404256232d70357c7f8446a20ba3466679f6a25939e0fe3ad83/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/169230503add734461e629d86dd31d7dbaa8d714c85a47be6aa2b89edf5e86c6/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/346cfbd7340274f66627fe302f8d20cb8fa38ba87a1c9a13e11c1b43d239e50f/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/7e6f90a9ac5402f32eea5c84f9c51c78f9b2248f090bbca53151d23df2fd1167/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/e1cf4a59e40de8a65ad4e04c13ade5ef3e6c52197cf4cfe903bdf088a2bb8ab9/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/32737a375a5a62238f47485ff728344356289d2f3826d0a44e45d712a3ad65a1/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/e49f6cb8565abc52a8bce8d092cfc892da64a80d76c9b7dd86794a5bed6d4898/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/2241dbbc718a8bf39b1ddb3b0f208e63357c10049f25a98c5a1f179e19ef5923/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/44a7cdcd2e55583cf95186c00c45b98102554ed67efc4efd7d7cc45764048705/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/f48f27283e65f1efc7e6e337ee8640727f92cb90ea0b4c23be5d775b9b2e400c/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/1be7669de360de9541dfccec9ed059b880bb71b1f9b4a161e0b721711e924d3f/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/e42497dd2a592c18628b288c16aeec4990be09e5f23548adda25d7ea2cf76709/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/ef8535d3030bc01b9fa4d3e299fbdf79e9275d14f2e0b75975c511ccb65025e5/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/a85b8897550a6b2a7231d6e93434b8620e8dff684d88b34e44e0382178b89898/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/072595eed49d530f160c70c614b1a589638a6cf715b40b448400d0c36ccf3924/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/2c55332f62e08f5d3872c5b9924c4a1bd9d9ff084e90e24ea38be4c98432974f/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/97f0329c59c044a5a6dc3f9dc00ee30342aa7ca03505c9ba70fa65979a520ea2/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/7f547b632b3f20acbaf7db510a215e1cd05948e3194ce37fc0467646fffd481d/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/51e50b669d1eb9367f3a3ddfa8673ef6082f3fcfde90b93d9d2eeadc1b68c7ea/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/c42c78f116ec573eb67be99eddc670eed80fdb676628171b8ac0b2154fb1ba62/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/cb4c5a00c24e5ede60babf1d2ff2cd5f24415637e8461c5dfe57379db7f6b797/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/c39ad5a43b40b597eaa47cd90c58b5d23b8a9ec18825f0a89be129b79303b62a/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/5648d9a5fb46de16d8f9f52cc8930121a5f60c039ab28b50461dd0b788fd0fc9/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/6e4499f4727cbfc6df958cee4226dad6abb36947d5fda6052b8e393ebba6b117/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/39e67a11191d626d89e5ff4ef6cae4da0cc48f7e26f5687062de8064de989b8c/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/64772b81b5233bf6d20511a8c99968d23a90c2ae71ad39af70aad177dddad77b/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/d4f9e4a206ee8458f2e505437e5836be3a08b321033a638119b99f2fe595fc83/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/e2f2c72b7d04cb59baa7201876f0598235039603a681671cb0348ceafd728eeb/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/2b87cf3c7de956b5d10ef752e162d031d7ba3b4f4a35e6a03b000d2418da42d3/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/020b666342ca489bb8745e5d55495adcdad88d2e3b120fd95064d09a3f679785/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/d32879bf9bd9666a75ea6e26706f38966de125c7c67cc5d6c87e932eddf49a92/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/878e0a4e00719e03d25f9e46bf47df03cc2b4591190d2a8186c918ce4c119e3c/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/d9567a206dad497994ee0564687a81cf5f8de44126e649f9c315fc846092f29b/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/d5f1e3a19c215a94a5544e7cfcd7f0dcb2e9d3fdaf6b36f53d92ae3e570ee72b/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/41057ff594f9e7e50d4dab97f057a9851b28ee8e2b2efcc9fe5451e8e5ecc6e7/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/0cd589cdd0e3bac2cbb772efb7621c672688d15ade96e1a8525c9f99c51292aa/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/d427fc0e639ecbd79bee3e74d808eaf4778e0353d2f037dff119e8105a30c616/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/bc6090a81bdfe4e6ab372c712b800443f1e3224a285563ac1f2df0968259f918/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/566c5d88542a361cc8b209dc1a110364a9c6bfcaa1c7de961a9002fd8aa2ebfc/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/103aed9daca272b1d386454e286891f9381f6f09c62da2ac4f3d4b700aee52b9/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/c8f2f6393d3d9a3c41753bcd550fcd0a23c047251839903710d0b6ec8296691f/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/06b37c6fda7b0f7fed9c5a540473cce218ff2534b92209e20ff3d89269f87f9d/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/f3e7a8c9208415ebbfc50031da236f2f272c2c53efa1056b5b4eb1f497447218/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-435:{mountpoint:/var/lib/containers/storage/overlay/aa6d9c59755fc319d51dd202b3388282ce68b160438160d31ca95ee96d7f28d1/merged major:0 minor:435 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/83dc7ce20979dbca3d188354c97bb18c2311f1eec4e0f48a603ee3172130edd6/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-439:{mountpoint:/var/lib/containers/storage/overlay/3257dae8e00ac24159cec2244aca1e43ff9557e8a4c078ec857a1e04f9225de2/merged major:0 minor:439 fsType:overlay blockSize:0} overlay_0-445:{mountpoint:/var/lib/containers/storage/overlay/949028e1922bbf236f680d62d36c3ed9d6dc6fc8205c4ce18ca70ce307c16de9/merged major:0 minor:445 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/04268eccf3df5cbae2750ea5c3a62d5328b0f911077a12a59d618f39ab8ff55e/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/c9babd724628e00dd81fca74d2fcd13e2112271864c62fc766bb3309c4289971/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-483:{mountpoint:/var/lib/containers/storage/overlay/416c78d812c3419a3d15b8218113c46fb1275e58b3a413b57a28fe06ce724ec5/merged major:0 minor:483 fsType:overlay blockSize:0} overlay_0-485:{mountpoint:/var/lib/containers/storage/overlay/c9c74b3f638bd95fb23deb72f2f94dc355ca2adced9663f71eebccc249cc317b/merged major:0 minor:485 fsType:overlay blockSize:0} overlay_0-487:{mountpoint:/var/lib/containers/storage/overlay/18a6e00e6fea25b900323ca84aea40dd5c3d1c969d4dca32c9bea397cb9a282a/merged major:0 minor:487 fsType:overlay blockSize:0} overlay_0-489:{mountpoint:/var/lib/containers/storage/overlay/6df43e6acadb16f2554214f901feed1b67ce8b3aac4b2472d161c4c300898a38/merged major:0 minor:489 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/adbc76f0205c8afb3abd17beeff9d4cc8ddc01db09baabb46e30d6b13ee1c607/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-494:{mountpoint:/var/lib/containers/storage/overlay/b11c1a85a90338c48c165e5d40e3d861c4244fd2c6ba30a8eb87ee3430eb35f5/merged major:0 minor:494 fsType:overlay blockSize:0} overlay_0-497:{mountpoint:/var/lib/containers/storage/overlay/157f715ec053a234932b94c6b57eb6f7da7d5232f0e9b3df246f7c951f0cb42f/merged major:0 minor:497 fsType:overlay blockSize:0} overlay_0-504:{mountpoint:/var/lib/containers/storage/overlay/36ac12d5d9c3989424a1d2ef2571aca321eb2acf8022a2720030b2dc4c437248/merged major:0 minor:504 fsType:overlay blockSize:0} overlay_0-517:{mountpoint:/var/lib/containers/storage/overlay/0ccedaeff27c08b13101883d17551f294c41632cb00c30ae907e1e2aecc6d46a/merged major:0 minor:517 fsType:overlay blockSize:0} overlay_0-519:{mountpoint:/var/lib/containers/storage/overlay/0c2365e8e817b934a670a6368ddaaaca837cf31a1323211d858f16efd437aca9/merged major:0 minor:519 fsType:overlay blockSize:0} overlay_0-521:{mountpoint:/var/lib/containers/storage/overlay/cad5142749fdc7f2d7b3100d8f8cb67bd3b8eedf9aba1d2f1cdc2a43fa5399b7/merged major:0 minor:521 fsType:overlay blockSize:0} overlay_0-526:{mountpoint:/var/lib/containers/storage/overlay/ec5ba98dc5d4e698f60bf07c21c2926f23cf8ae06d3b5befed580ae111faddac/merged major:0 minor:526 fsType:overlay blockSize:0} overlay_0-529:{mountpoint:/var/lib/containers/storage/overlay/e3db1e89e1930b6c82b34ce50152cb8e512daac5bb4baf5793be5cbcf2f4a908/merged major:0 minor:529 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/cab5711e5b6fd5d140fe92535e57c67e644c076d25a0d32bec58a3fca01ff09a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ff32de2c5184a72da5351f836eaa234dfe5a493ec9ef0c3760657236aa5c2cf0/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/242be3154df7ccf3d4236871e7ccd5dea00e45293753785240b8e81fb3bcc4db/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/037b0294eab5c72d98293bd7bf91f8946a482b321cc8f6a55b38fd09160245a1/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/52cebe4759fe5a26d6428ee2d537b61f7fb11911600d7b8bed1e89ae4ed6e782/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/d2793843b6b8239973978b761755811fb628d9c27e232faf07f13d912c6afe3a/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/106c9b937dafe9e9de46131a7d945b991a14c22adaa384a7d44d4cf65c99b117/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/2ed9c78d6a2520ae44c2fe788e24f39099668857ecb5944021f332bb88a72115/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/cb3f5c13e6a3d4dc6b32881a8bbaa0195383b7202d252305920c6d63beb57226/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 04 01:10:50.061153 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.060447 5995 manager.go:217] Machine: {Timestamp:2026-11-04 01:10:50.059546298 +0000 UTC m=+0.091279682 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3194533 MemoryCapacity:33164345344 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e59fb50382bd07caaa74aa1274482 SystemUUID:ec2e59fb-5038-2bd0-7caa-a74aa1274482 BootID:c21fe939-1ab8-4b02-8f68-f34af617cb1f Filesystems:[{Device:/var/lib/kubelet/pods/5cecaa47-44ef-440e-80eb-45230fe3edc0/volumes/kubernetes.io~projected/kube-api-access-d94r5 DeviceMajor:0 DeviceMinor:197 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7aa61c-2789-49ed-989d-fa77fed0ff0f/volumes/kubernetes.io~projected/kube-api-access-gsq8f DeviceMajor:0 DeviceMinor:217 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/c2aa236a-4ff4-4b39-90ff-c782721444ee/volumes/kubernetes.io~projected/kube-api-access-cx7dg DeviceMajor:0 DeviceMinor:234 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29/volumes/kubernetes.io~projected/kube-api-access-827hl DeviceMajor:0 DeviceMinor:307 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6c384c5ee28e82f224c188c97a0325dd539c534432b42c0bf543d8adaaa8a9e/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12b8e2e7d0b1cfe4c92e5f13d3fe78740de4021fc7f21746a5bee63bba5a4ec4/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57632676aacede02ef6f25328a842b6cdddff2f9f7290826b4767f5643f043f0/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c93d8fe3521e9d82ee6651bc3b234e56a2e47a042a2b191d87d595990d2def2/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf/volumes/kubernetes.io~projected/kube-api-access-chq5x DeviceMajor:0 DeviceMinor:345 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/3ae06262-6e59-4775-bedc-5305ace1fae6/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:475 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-497 DeviceMajor:0 DeviceMinor:497 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8/volumes/kubernetes.io~projected/kube-api-access-pm8jv DeviceMajor:0 DeviceMinor:61 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/f2df8dc5-6403-4827-b6d6-9c209c595e05/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:206 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/ae64edd1-43dc-406d-bcad-f7f34bb1f991/volumes/kubernetes.io~projected/kube-api-access-v22h7 DeviceMajor:0 DeviceMinor:230 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/ca77e439-49cb-4ef0-9951-9564ab28ed3e/volumes/kubernetes.io~projected/kube-api-access-lk265 DeviceMajor:0 DeviceMinor:237 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/8ddedc2c-735e-41e5-a5a9-3386ed890ee1/volumes/kubernetes.io~projected/kube-api-access-kxsvl DeviceMajor:0 DeviceMinor:330 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7f25d5d02c71d60f4299230aed43c4c7113dfb0a34777bf62d08ea66c862290/userdata/shm DeviceMajor:0 DeviceMinor:477 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c35ab28fb00962801a8fe34285bf6942fc29ad7360a9e7604f957437f7d950f/userdata/shm DeviceMajor:0 DeviceMinor:464 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-483 DeviceMajor:0 DeviceMinor:483 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03f3058b-0663-48a7-ab67-a41bfe5a2c57/volumes/kubernetes.io~projected/kube-api-access-rlf5p DeviceMajor:0 DeviceMinor:60 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98da5efe82e0b952f83c1d1541ad31ac5ab87b519f7a25be001d0ab695443267/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/acf4fc0ba80c720332e88d492e223ff678c13b3e7604db81bd64148c02b40836/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd605a92-f88c-40ea-bde7-53c2da66a576/volumes/kubernetes.io~projected/kube-api-access-9tf5p DeviceMajor:0 DeviceMinor:201 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-445 DeviceMajor:0 DeviceMinor:445 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/13d5686dc21a2275c049fb9f51447a8f8a3652ae89ee45e5e95bc5efead89a01/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/c5b0a3f1-5189-4572-9765-20eb3960fbae/volumes/kubernetes.io~projected/kube-api-access-245fq DeviceMajor:0 DeviceMinor:65 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2df8dc5-6403-4827-b6d6-9c209c595e05/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:207 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03f3058b-0663-48a7-ab67-a41bfe5a2c57/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/5b8947a9-b2f0-4de2-a1b8-58342b6b3240/volumes/kubernetes.io~secret/networking-console-plugin-cert DeviceMajor:0 DeviceMinor:375 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82966365-6bec-467d-9279-12ac45dd01e2/volumes/kubernetes.io~projected/kube-api-access-fpmgk DeviceMajor:0 DeviceMinor:66 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89bd09a4-6d44-49ae-9bfa-837e2e5bf915/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-526 DeviceMajor:0 DeviceMinor:526 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2df8dc5-6403-4827-b6d6-9c209c595e05/volumes/kubernetes.io~projected/kube-api-access-jl9gz DeviceMajor:0 DeviceMinor:215 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/5d1de04c-5111-415b-bdcc-20dfa9d0e6f1/volumes/kubernetes.io~projected/kube-api-access-j8rgj DeviceMajor:0 DeviceMinor:456 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb55ed6-b225-4db5-a7fb-66a10868befc/volumes/kubernetes.io~projected/kube-api-access-slglc DeviceMajor:0 DeviceMinor:64 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18cf5ce0a486bf5b432b3279552d68791ee83754ac6cc2a8bf87576938774556/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8b2b155-18ef-40a1-b87b-a6e44e130d65/volumes/kubernetes.io~projected/kube-api-access-9dxcj DeviceMajor:0 DeviceMinor:202 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/6250a398-c824-4c6e-9083-c7b0df543700/volumes/kubernetes.io~projected/kube-api-access-p78dk DeviceMajor:0 DeviceMinor:233 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b44a2a16-b00a-4113-b0ab-5cdcb955b325/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5bc62a7e61ebb236bb4631d93bab8331783661fd6a433c1d3f2744be9b012835/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/83ffd8448ded9b9ec3ef6209f1bf5d4b96993fb173ab544e4321f7611f36e6ce/userdata/shm DeviceMajor:0 DeviceMinor:320 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76b3d30e17580e47f2e408a0debe8067481a8f8432dc342f8f86a0b6cc06020d/userdata/shm DeviceMajor:0 DeviceMinor:376 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f80bab462e52ddc7607885b1de473b4caed00db64ad6f17487d337795134196/userdata/shm DeviceMajor:0 DeviceMinor:379 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98172af2c8dfd62512835ca75a3182e74da317adbe908f18a05b0e0c39af5bfa/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2aa236a-4ff4-4b39-90ff-c782721444ee/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:223 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/e96cde4e-0238-48e6-b343-a0592f6e159e/volumes/kubernetes.io~projected/kube-api-access-ks2wj DeviceMajor:0 DeviceMinor:236 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2df8dc5-6403-4827-b6d6-9c209c595e05/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:371 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f47617b789c3b3cb03076d6552d01b677d953d5f1021e70bb844f965e7f17cf/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9544ea47f6ea8e8786d664c14ef2178951957028bd8ff037e155ae0b8a532e82/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-439 DeviceMajor:0 DeviceMinor:439 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:459 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/89bd09a4-6d44-49ae-9bfa-837e2e5bf915/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a23c835a-aacb-4f92-a2cd-799f5e870b85/volumes/kubernetes.io~projected/kube-api-access-kpfww DeviceMajor:0 DeviceMinor:203 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-435 DeviceMajor:0 DeviceMinor:435 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-487 DeviceMajor:0 DeviceMinor:487 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-504 DeviceMajor:0 DeviceMinor:504 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03f3058b-0663-48a7-ab67-a41bfe5a2c57/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5/volumes/kubernetes.io~projected/kube-api-access-ltmvj DeviceMajor:0 DeviceMinor:62 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5cecaa47-44ef-440e-80eb-45230fe3edc0/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0750204d2e8b59273e5d36bb112a2603bbe424206ef65dd634bb6a7a3efa927d/userdata/shm DeviceMajor:0 DeviceMinor:369 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29855f357f0133564c0d8111980f94532023567cc7f628e722b906f68c1c6a2c/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ecbf13bab42e195b5371678969094ac07ed75be0035a7afed257df0aadbe0af/userdata/shm DeviceMajor:0 DeviceMinor:457 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/0f50bc9b-91da-4d92-8553-c9f9476665e4/volumes/kubernetes.io~projected/kube-api-access-zhhtl DeviceMajor:0 DeviceMinor:59 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b44a2a16-b00a-4113-b0ab-5cdcb955b325/volumes/kubernetes.io~projected/kube-api-access-2pbbl DeviceMajor:0 DeviceMinor:216 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/ae64edd1-43dc-406d-bcad-f7f34bb1f991/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:222 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2dd755b06f3e13f4f7881c0031f294ae2d40a05f702199edab0368f01ee59c6/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/a7991c77-7106-4c73-8812-ff6d4c06b666/volumes/kubernetes.io~projected/kube-api-access-b7mxs DeviceMajor:0 DeviceMinor:214 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6fbf21c0ddf595ca72a79c5e68ff7f45cd411422580b519df444546ce79f4f6/userdata/shm DeviceMajor:0 DeviceMinor:433 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-489 DeviceMajor:0 DeviceMinor:489 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e96cde4e-0238-48e6-b343-a0592f6e159e/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:378 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:461 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65ee99eef508daea64d5d7e15539739e09ad12a1e442f4067af4dc52aabf6ba0/userdata/shm DeviceMajor:0 DeviceMinor:510 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582172672 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97eefcf4cb3d26c1db3a3ea4102ab8d0e9226c78b7451d734136ca782bf547f8/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8df2f8562b1d8be2e8110165044fe414fe196d1fa649d8d0218c9d4dce16a11/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/251e21ccd12219fcf62ea2f806f78dc2c3356ebc03f66d402fbcd980dd360e8a/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cfb374093c7ff29f6badcdd2465ae551ee208b14450e298c52bc5a9384416355/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30e0df48fcd09cdce51e26071a7daec8132477b4ae20bb7af8f568959ad13341/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/747ed7ed-d05b-434a-8360-acc2292c2257/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:450 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe/volumes/kubernetes.io~projected/kube-api-access-xlvzh DeviceMajor:0 DeviceMinor:462 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-517 DeviceMajor:0 DeviceMinor:517 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89bd09a4-6d44-49ae-9bfa-837e2e5bf915/volumes/kubernetes.io~projected/kube-api-access-bk7ck DeviceMajor:0 DeviceMinor:67 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75c5b017-8f43-41dc-b79b-0353de290289/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls DeviceMajor:0 DeviceMinor:364 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/44d7b0a2497bef282b17ba2e0e4b41b6cca26c8d3a27b5de2f18c3746254701d/userdata/shm DeviceMajor:0 DeviceMinor:512 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/c3d08d00-e371-4793-955e-c5b51eef7879/volumes/kubernetes.io~projected/kube-api-access-v8xv5 DeviceMajor:0 DeviceMinor:63 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75c5b017-8f43-41dc-b79b-0353de290289/volumes/kubernetes.io~projected/kube-api-access-dm4k6 DeviceMajor:0 DeviceMinor:204 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a06edce6-baf3-40bc-9d0e-a6e71c393165/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2aa236a-4ff4-4b39-90ff-c782721444ee/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:221 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c60eb5db-ec53-40bb-9a72-c712cec7c32c/volumes/kubernetes.io~projected/kube-api-access-4b5fm DeviceMajor:0 DeviceMinor:335 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-521 DeviceMajor:0 DeviceMinor:521 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2aa236a-4ff4-4b39-90ff-c782721444ee/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:232 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:344 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36d411089d38ab9155869fc5fc027a051a82de1059856529daf993a1c77e0a67/userdata/shm DeviceMajor:0 DeviceMinor:256 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:460 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7573b3c8-fd6e-41c5-9de4-9c441f2ce026/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a02e524282e41a87c6ab6ad18b28a081e82927589533c29c68df9f4ea7a8a94c/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a889499cabdba0d2fd9a2ef3f5313b44a484207a29a7b6f6f795fcf287ea34be/userdata/shm DeviceMajor:0 DeviceMinor:454 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:463 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/875c7ad36433ca2510d1f9bc776d1abf6a9ee2bef42bae468edd1159927d291a/userdata/shm DeviceMajor:0 DeviceMinor:479 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-519 DeviceMajor:0 DeviceMinor:519 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7573b3c8-fd6e-41c5-9de4-9c441f2ce026/volumes/kubernetes.io~projected/kube-api-access-dp5d9 DeviceMajor:0 DeviceMinor:213 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86a37a601387a008f2c0ddd8f57e96ab06eb284bd0203263892fec8bfbd4aec4/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a82c2a760402ddb4b1ed8bacf6330c70ac99acd53cf7e314cc7d25d2280272e/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef7aa61c-2789-49ed-989d-fa77fed0ff0f/volumes/kubernetes.io~secret/samples-operator-tls DeviceMajor:0 DeviceMinor:372 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/c5dd40be-e515-4fe8-be55-b93f5b97973e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:429 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/ca77e439-49cb-4ef0-9951-9564ab28ed3e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:235 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:470 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69a81c3f70096360826bcc68061e4a63af85de5c099c0524fa9ba022f6671d82/userdata/shm DeviceMajor:0 DeviceMinor:481 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/c5dd40be-e515-4fe8-be55-b93f5b97973e/volumes/kubernetes.io~projected/kube-api-access-2tgzg DeviceMajor:0 DeviceMinor:306 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42f02121-70f2-43d1-be9d-2a81425ec97d/volumes/kubernetes.io~projected/kube-api-access-sxlfx DeviceMajor:0 DeviceMinor:476 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-485 DeviceMajor:0 DeviceMinor:485 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-494 DeviceMajor:0 DeviceMinor:494 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2aa236a-4ff4-4b39-90ff-c782721444ee/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:503 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-529 DeviceMajor:0 DeviceMinor:529 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/738361f938aa7c6d4c807186a4dced11bcc6dbea29c8580aeff49fc3f28ff4f2/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0750204d2e8b592 MacAddress:b6:0c:b9:a0:0c:7d Speed:10000 Mtu:8901} {Name:1f80bab462e52dd MacAddress:fa:20:88:be:73:26 Speed:10000 Mtu:8901} {Name:251e21ccd12219f MacAddress:5a:5a:37:ef:92:aa Speed:10000 Mtu:8901} {Name:29855f357f01335 MacAddress:22:dd:85:e7:bf:17 Speed:10000 Mtu:8901} {Name:30e0df48fcd09cd MacAddress:9a:e7:2d:bb:40:8e Speed:10000 Mtu:8901} {Name:36d411089d38ab9 MacAddress:12:6a:f0:9b:c7:22 Speed:10000 Mtu:8901} {Name:44d7b0a2497bef2 MacAddress:ea:e7:7b:b1:e0:1a Speed:10000 Mtu:8901} {Name:4a82c2a760402dd MacAddress:16:d6:c6:f9:87:30 Speed:10000 Mtu:8901} {Name:5bc62a7e61ebb23 MacAddress:8a:62:6a:39:51:17 Speed:10000 Mtu:8901} {Name:5ecbf13bab42e19 MacAddress:5e:65:65:51:12:4c Speed:10000 Mtu:8901} {Name:65ee99eef508dae MacAddress:b6:91:68:c9:77:29 Speed:10000 Mtu:8901} {Name:69a81c3f7009636 MacAddress:6e:5d:e6:fc:9a:72 Speed:10000 Mtu:8901} {Name:6c93d8fe3521e9d MacAddress:fa:5b:cd:df:ba:30 Speed:10000 Mtu:8901} {Name:76b3d30e17580e4 MacAddress:a2:50:e1:7e:79:9f Speed:10000 Mtu:8901} {Name:83ffd8448ded9b9 MacAddress:1a:32:2e:3e:21:fd Speed:10000 Mtu:8901} {Name:86a37a601387a00 MacAddress:da:12:1a:42:60:fd Speed:10000 Mtu:8901} {Name:875c7ad36433ca2 MacAddress:86:77:ee:02:10:f1 Speed:10000 Mtu:8901} {Name:8c35ab28fb00962 MacAddress:ce:16:a6:28:76:14 Speed:10000 Mtu:8901} {Name:9544ea47f6ea8e8 MacAddress:6e:19:b1:06:57:8a Speed:10000 Mtu:8901} {Name:a02e524282e41a8 MacAddress:fa:6d:f4:33:20:dd Speed:10000 Mtu:8901} {Name:a889499cabdba0d MacAddress:32:dc:74:3f:ac:3b Speed:10000 Mtu:8901} {Name:a8df2f8562b1d8b MacAddress:da:34:b7:70:95:2d Speed:10000 Mtu:8901} {Name:b7f25d5d02c71d6 MacAddress:6e:8e:b5:35:ab:2a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:da:1b:da:27 Speed:0 Mtu:9001} {Name:br-int MacAddress:ce:e7:c3:58:f1:cf Speed:0 Mtu:8901} {Name:c6c384c5ee28e82 MacAddress:9a:0f:4b:af:53:c9 Speed:10000 Mtu:8901} {Name:cfb374093c7ff29 MacAddress:a6:2d:8a:3d:b1:0b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:da:1b:da:27 Speed:0 Mtu:9001} {Name:f2dd755b06f3e13 MacAddress:9a:c2:14:d3:3e:aa Speed:10000 Mtu:8901} {Name:f6fbf21c0ddf595 MacAddress:f6:3e:68:db:da:b5 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:06:c7:c9:61:93:7a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:82:7c:36:62:5b:92 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164345344 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 01:10:50.061476 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.061160 5995 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 01:10:50.061476 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.061225 5995 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 01:10:50.061562 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.061532 5995 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 01:10:50.061688 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.061563 5995 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-179.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 01:10:50.061728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.061692 5995 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 01:10:50.061728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.061700 5995 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 01:10:50.061728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.061711 5995 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 01:10:50.061728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.061723 5995 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 01:10:50.061827 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.061821 5995 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 01:10:50.062016 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.062006 5995 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 01:10:50.062131 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.062123 5995 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 01:10:50.062172 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.062163 5995 kubelet.go:493] "Attempting to sync node with API server" Nov 04 01:10:50.062210 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.062177 5995 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 01:10:50.062210 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.062187 5995 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 01:10:50.062210 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.062204 5995 kubelet.go:405] "Adding apiserver pod source" Nov 04 01:10:50.062309 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.062213 5995 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 01:10:50.063319 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.063302 5995 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 01:10:50.063363 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.063322 5995 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 01:10:50.063953 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.063938 5995 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 01:10:50.076527 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.076504 5995 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 01:10:50.076527 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.076523 5995 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 01:10:50.076660 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.076639 5995 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 04 01:10:50.076868 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.076855 5995 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 01:10:50.076899 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.076887 5995 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 01:10:50.077025 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.077015 5995 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 01:10:50.077055 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.077037 5995 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 01:10:50.077055 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.077045 5995 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 01:10:50.077055 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.077055 5995 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 01:10:50.077127 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.077062 5995 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 01:10:50.077127 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.077068 5995 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 01:10:50.077127 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.077074 5995 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 01:10:50.077127 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.077083 5995 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 01:10:50.077127 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.077094 5995 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 01:10:50.077127 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.077100 5995 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 01:10:50.077127 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.077109 5995 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 01:10:50.077127 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.077119 5995 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 01:10:50.077355 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.077145 5995 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 01:10:50.077355 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.077154 5995 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 01:10:50.080898 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.080884 5995 server.go:1263] "Started kubelet" Nov 04 01:10:50.081452 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.080949 5995 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 01:10:50.081452 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.081069 5995 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 01:10:50.081452 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.081444 5995 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 01:10:50.081528 ip-10-0-138-179 systemd[1]: Started Kubernetes Kubelet. Nov 04 01:10:50.082172 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.081760 5995 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 01:10:50.082501 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.082484 5995 server.go:317] "Adding debug handlers to kubelet server" Nov 04 01:10:50.089716 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.089555 5995 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 01:10:50.089716 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.089610 5995 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 01:10:50.089716 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.089659 5995 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 01:10:50.089716 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.089672 5995 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 01:10:50.089716 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.089656 5995 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 01:04:38 +0000 UTC" deadline="2028-06-02 23:22:43.57008055 +0000 UTC" Nov 04 01:10:50.089716 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.089691 5995 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13846h11m53.480392136s" Nov 04 01:10:50.089716 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:10:50.089703 5995 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-179.us-west-2.compute.internal\" not found" Nov 04 01:10:50.090037 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.089820 5995 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 01:10:50.091545 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.091521 5995 factory.go:55] Registering systemd factory Nov 04 01:10:50.091649 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.091550 5995 factory.go:223] Registration of the systemd container factory successfully Nov 04 01:10:50.092095 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.092009 5995 factory.go:153] Registering CRI-O factory Nov 04 01:10:50.092095 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.092029 5995 factory.go:223] Registration of the crio container factory successfully Nov 04 01:10:50.092095 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.092078 5995 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 01:10:50.092095 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.092097 5995 factory.go:103] Registering Raw factory Nov 04 01:10:50.092336 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.092117 5995 manager.go:1199] Started watching for new ooms in manager Nov 04 01:10:50.092688 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.092644 5995 manager.go:319] Starting recovery of all containers Nov 04 01:10:50.094744 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.094674 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe" volumeName="kubernetes.io/empty-dir/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-ca-trust-extracted" seLinuxMountContext="" Nov 04 01:10:50.094744 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.094738 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe" volumeName="kubernetes.io/secret/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-installation-pull-secrets" seLinuxMountContext="" Nov 04 01:10:50.094913 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.094763 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89bd09a4-6d44-49ae-9bfa-837e2e5bf915" volumeName="kubernetes.io/secret/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-ovn-node-metrics-cert" seLinuxMountContext="" Nov 04 01:10:50.094913 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.094777 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca77e439-49cb-4ef0-9951-9564ab28ed3e" volumeName="kubernetes.io/secret/ca77e439-49cb-4ef0-9951-9564ab28ed3e-cert" seLinuxMountContext="" Nov 04 01:10:50.094913 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.094789 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2df8dc5-6403-4827-b6d6-9c209c595e05" volumeName="kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle" seLinuxMountContext="" Nov 04 01:10:50.094913 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.094814 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" volumeName="kubernetes.io/projected/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-kube-api-access-pm8jv" seLinuxMountContext="" Nov 04 01:10:50.094913 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.094827 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5" volumeName="kubernetes.io/projected/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-kube-api-access-ltmvj" seLinuxMountContext="" Nov 04 01:10:50.094913 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.094840 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03f3058b-0663-48a7-ab67-a41bfe5a2c57" volumeName="kubernetes.io/empty-dir/03f3058b-0663-48a7-ab67-a41bfe5a2c57-tmp" seLinuxMountContext="" Nov 04 01:10:50.094913 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.094865 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ae06262-6e59-4775-bedc-5305ace1fae6" volumeName="kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret" seLinuxMountContext="" Nov 04 01:10:50.094913 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.094889 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7573b3c8-fd6e-41c5-9de4-9c441f2ce026" volumeName="kubernetes.io/projected/7573b3c8-fd6e-41c5-9de4-9c441f2ce026-kube-api-access-dp5d9" seLinuxMountContext="" Nov 04 01:10:50.094913 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.094902 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89bd09a4-6d44-49ae-9bfa-837e2e5bf915" volumeName="kubernetes.io/configmap/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-env-overrides" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.094926 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2aa236a-4ff4-4b39-90ff-c782721444ee" volumeName="kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-trusted-ca" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.094939 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3d08d00-e371-4793-955e-c5b51eef7879" volumeName="kubernetes.io/projected/c3d08d00-e371-4793-955e-c5b51eef7879-kube-api-access-v8xv5" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.094951 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e96cde4e-0238-48e6-b343-a0592f6e159e" volumeName="kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.094976 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b8947a9-b2f0-4de2-a1b8-58342b6b3240" volumeName="kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.094989 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cecaa47-44ef-440e-80eb-45230fe3edc0" volumeName="kubernetes.io/empty-dir/5cecaa47-44ef-440e-80eb-45230fe3edc0-tmp" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095001 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cecaa47-44ef-440e-80eb-45230fe3edc0" volumeName="kubernetes.io/secret/5cecaa47-44ef-440e-80eb-45230fe3edc0-serving-cert" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095048 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe" volumeName="kubernetes.io/projected/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-registry-tls" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095061 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5" volumeName="kubernetes.io/configmap/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-cni-binary-copy" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095073 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae64edd1-43dc-406d-bcad-f7f34bb1f991" volumeName="kubernetes.io/configmap/ae64edd1-43dc-406d-bcad-f7f34bb1f991-trusted-ca" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095096 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae64edd1-43dc-406d-bcad-f7f34bb1f991" volumeName="kubernetes.io/projected/ae64edd1-43dc-406d-bcad-f7f34bb1f991-kube-api-access-v22h7" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095108 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2aa236a-4ff4-4b39-90ff-c782721444ee" volumeName="kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-kube-api-access-cx7dg" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095120 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d1de04c-5111-415b-bdcc-20dfa9d0e6f1" volumeName="kubernetes.io/projected/5d1de04c-5111-415b-bdcc-20dfa9d0e6f1-kube-api-access-j8rgj" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095156 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3d08d00-e371-4793-955e-c5b51eef7879" volumeName="kubernetes.io/configmap/c3d08d00-e371-4793-955e-c5b51eef7879-cni-sysctl-allowlist" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095171 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2df8dc5-6403-4827-b6d6-9c209c595e05" volumeName="kubernetes.io/projected/f2df8dc5-6403-4827-b6d6-9c209c595e05-kube-api-access-jl9gz" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095197 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2df8dc5-6403-4827-b6d6-9c209c595e05" volumeName="kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-default-certificate" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095242 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb55ed6-b225-4db5-a7fb-66a10868befc" volumeName="kubernetes.io/projected/ffb55ed6-b225-4db5-a7fb-66a10868befc-kube-api-access-slglc" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095256 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe" volumeName="kubernetes.io/configmap/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-trusted-ca" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095272 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe" volumeName="kubernetes.io/projected/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-bound-sa-token" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095309 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2f6e24e-f98d-4b9a-8d51-080c2c38cf29" volumeName="kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-service-ca-bundle" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095322 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae64edd1-43dc-406d-bcad-f7f34bb1f991" volumeName="kubernetes.io/configmap/ae64edd1-43dc-406d-bcad-f7f34bb1f991-config" seLinuxMountContext="" Nov 04 01:10:50.095326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095334 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2aa236a-4ff4-4b39-90ff-c782721444ee" volumeName="kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-installation-pull-secrets" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095346 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3d08d00-e371-4793-955e-c5b51eef7879" volumeName="kubernetes.io/configmap/c3d08d00-e371-4793-955e-c5b51eef7879-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095359 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5dd40be-e515-4fe8-be55-b93f5b97973e" volumeName="kubernetes.io/empty-dir/c5dd40be-e515-4fe8-be55-b93f5b97973e-data-volume" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095372 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2f6e24e-f98d-4b9a-8d51-080c2c38cf29" volumeName="kubernetes.io/empty-dir/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-archives-path" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095386 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2aa236a-4ff4-4b39-90ff-c782721444ee" volumeName="kubernetes.io/empty-dir/c2aa236a-4ff4-4b39-90ff-c782721444ee-ca-trust-extracted" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095398 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2aa236a-4ff4-4b39-90ff-c782721444ee" volumeName="kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095409 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4093a5e-62e9-4f40-b3ad-15453e4f1bcf" volumeName="kubernetes.io/secret/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf-signing-key" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095437 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2f6e24e-f98d-4b9a-8d51-080c2c38cf29" volumeName="kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-trusted-ca-bundle" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095459 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7aa61c-2789-49ed-989d-fa77fed0ff0f" volumeName="kubernetes.io/projected/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-kube-api-access-gsq8f" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095470 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b44a2a16-b00a-4113-b0ab-5cdcb955b325" volumeName="kubernetes.io/projected/b44a2a16-b00a-4113-b0ab-5cdcb955b325-kube-api-access-2pbbl" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095484 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f50bc9b-91da-4d92-8553-c9f9476665e4" volumeName="kubernetes.io/configmap/0f50bc9b-91da-4d92-8553-c9f9476665e4-serviceca" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095497 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f50bc9b-91da-4d92-8553-c9f9476665e4" volumeName="kubernetes.io/projected/0f50bc9b-91da-4d92-8553-c9f9476665e4-kube-api-access-zhhtl" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095521 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b8947a9-b2f0-4de2-a1b8-58342b6b3240" volumeName="kubernetes.io/configmap/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-nginx-conf" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095532 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe" volumeName="kubernetes.io/projected/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-kube-api-access-xlvzh" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095543 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3d08d00-e371-4793-955e-c5b51eef7879" volumeName="kubernetes.io/configmap/c3d08d00-e371-4793-955e-c5b51eef7879-cni-binary-copy" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095555 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7aa61c-2789-49ed-989d-fa77fed0ff0f" volumeName="kubernetes.io/configmap/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-config" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095569 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ddedc2c-735e-41e5-a5a9-3386ed890ee1" volumeName="kubernetes.io/projected/8ddedc2c-735e-41e5-a5a9-3386ed890ee1-kube-api-access-kxsvl" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095581 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae64edd1-43dc-406d-bcad-f7f34bb1f991" volumeName="kubernetes.io/secret/ae64edd1-43dc-406d-bcad-f7f34bb1f991-serving-cert" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095594 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cecaa47-44ef-440e-80eb-45230fe3edc0" volumeName="kubernetes.io/empty-dir/5cecaa47-44ef-440e-80eb-45230fe3edc0-snapshots" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095605 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cecaa47-44ef-440e-80eb-45230fe3edc0" volumeName="kubernetes.io/projected/5cecaa47-44ef-440e-80eb-45230fe3edc0-kube-api-access-d94r5" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095617 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7573b3c8-fd6e-41c5-9de4-9c441f2ce026" volumeName="kubernetes.io/secret/7573b3c8-fd6e-41c5-9de4-9c441f2ce026-serving-cert" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095628 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75c5b017-8f43-41dc-b79b-0353de290289" volumeName="kubernetes.io/configmap/75c5b017-8f43-41dc-b79b-0353de290289-telemetry-config" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095641 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75c5b017-8f43-41dc-b79b-0353de290289" volumeName="kubernetes.io/projected/75c5b017-8f43-41dc-b79b-0353de290289-kube-api-access-dm4k6" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095654 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82966365-6bec-467d-9279-12ac45dd01e2" volumeName="kubernetes.io/projected/82966365-6bec-467d-9279-12ac45dd01e2-kube-api-access-fpmgk" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095677 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a06edce6-baf3-40bc-9d0e-a6e71c393165" volumeName="kubernetes.io/configmap/a06edce6-baf3-40bc-9d0e-a6e71c393165-konnectivity-ca" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095698 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2aa236a-4ff4-4b39-90ff-c782721444ee" volumeName="kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-certificates" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095711 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d1de04c-5111-415b-bdcc-20dfa9d0e6f1" volumeName="kubernetes.io/empty-dir/5d1de04c-5111-415b-bdcc-20dfa9d0e6f1-tmp" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095722 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89bd09a4-6d44-49ae-9bfa-837e2e5bf915" volumeName="kubernetes.io/configmap/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-ovnkube-script-lib" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095734 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4093a5e-62e9-4f40-b3ad-15453e4f1bcf" volumeName="kubernetes.io/projected/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf-kube-api-access-chq5x" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095746 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5dd40be-e515-4fe8-be55-b93f5b97973e" volumeName="kubernetes.io/projected/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-api-access-2tgzg" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095758 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd605a92-f88c-40ea-bde7-53c2da66a576" volumeName="kubernetes.io/projected/cd605a92-f88c-40ea-bde7-53c2da66a576-kube-api-access-9tf5p" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095770 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8" volumeName="kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095781 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03f3058b-0663-48a7-ab67-a41bfe5a2c57" volumeName="kubernetes.io/empty-dir/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-tuned" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095793 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cecaa47-44ef-440e-80eb-45230fe3edc0" volumeName="kubernetes.io/configmap/5cecaa47-44ef-440e-80eb-45230fe3edc0-trusted-ca-bundle" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095807 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75c5b017-8f43-41dc-b79b-0353de290289" volumeName="kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095819 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89bd09a4-6d44-49ae-9bfa-837e2e5bf915" volumeName="kubernetes.io/configmap/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-ovnkube-config" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095830 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2f6e24e-f98d-4b9a-8d51-080c2c38cf29" volumeName="kubernetes.io/projected/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-kube-api-access-827hl" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095842 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e96cde4e-0238-48e6-b343-a0592f6e159e" volumeName="kubernetes.io/projected/e96cde4e-0238-48e6-b343-a0592f6e159e-kube-api-access-ks2wj" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095855 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a06edce6-baf3-40bc-9d0e-a6e71c393165" volumeName="kubernetes.io/secret/a06edce6-baf3-40bc-9d0e-a6e71c393165-agent-certs" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095867 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5dd40be-e515-4fe8-be55-b93f5b97973e" volumeName="kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095880 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2df8dc5-6403-4827-b6d6-9c209c595e05" volumeName="kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-stats-auth" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095892 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a23c835a-aacb-4f92-a2cd-799f5e870b85" volumeName="kubernetes.io/projected/a23c835a-aacb-4f92-a2cd-799f5e870b85-kube-api-access-kpfww" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095904 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca77e439-49cb-4ef0-9951-9564ab28ed3e" volumeName="kubernetes.io/projected/ca77e439-49cb-4ef0-9951-9564ab28ed3e-kube-api-access-lk265" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095915 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e96cde4e-0238-48e6-b343-a0592f6e159e" volumeName="kubernetes.io/configmap/e96cde4e-0238-48e6-b343-a0592f6e159e-config-volume" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095927 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef7aa61c-2789-49ed-989d-fa77fed0ff0f" volumeName="kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095938 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2df8dc5-6403-4827-b6d6-9c209c595e05" volumeName="kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095953 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03f3058b-0663-48a7-ab67-a41bfe5a2c57" volumeName="kubernetes.io/projected/03f3058b-0663-48a7-ab67-a41bfe5a2c57-kube-api-access-rlf5p" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095965 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6250a398-c824-4c6e-9083-c7b0df543700" volumeName="kubernetes.io/projected/6250a398-c824-4c6e-9083-c7b0df543700-kube-api-access-p78dk" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095976 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7573b3c8-fd6e-41c5-9de4-9c441f2ce026" volumeName="kubernetes.io/configmap/7573b3c8-fd6e-41c5-9de4-9c441f2ce026-config" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095987 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe" volumeName="kubernetes.io/configmap/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-registry-certificates" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.095999 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b44a2a16-b00a-4113-b0ab-5cdcb955b325" volumeName="kubernetes.io/secret/b44a2a16-b00a-4113-b0ab-5cdcb955b325-serving-cert" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096012 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2aa236a-4ff4-4b39-90ff-c782721444ee" volumeName="kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-bound-sa-token" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096024 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5b0a3f1-5189-4572-9765-20eb3960fbae" volumeName="kubernetes.io/configmap/c5b0a3f1-5189-4572-9765-20eb3960fbae-iptables-alerter-script" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096035 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c60eb5db-ec53-40bb-9a72-c712cec7c32c" volumeName="kubernetes.io/projected/c60eb5db-ec53-40bb-9a72-c712cec7c32c-kube-api-access-4b5fm" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096047 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cecaa47-44ef-440e-80eb-45230fe3edc0" volumeName="kubernetes.io/configmap/5cecaa47-44ef-440e-80eb-45230fe3edc0-service-ca-bundle" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096059 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe" volumeName="kubernetes.io/secret/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-image-registry-private-configuration" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096072 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89bd09a4-6d44-49ae-9bfa-837e2e5bf915" volumeName="kubernetes.io/projected/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-kube-api-access-bk7ck" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096084 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b44a2a16-b00a-4113-b0ab-5cdcb955b325" volumeName="kubernetes.io/configmap/b44a2a16-b00a-4113-b0ab-5cdcb955b325-config" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096096 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e96cde4e-0238-48e6-b343-a0592f6e159e" volumeName="kubernetes.io/empty-dir/e96cde4e-0238-48e6-b343-a0592f6e159e-tmp-dir" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096107 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb55ed6-b225-4db5-a7fb-66a10868befc" volumeName="kubernetes.io/empty-dir/ffb55ed6-b225-4db5-a7fb-66a10868befc-tmp-dir" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096118 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42f02121-70f2-43d1-be9d-2a81425ec97d" volumeName="kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096131 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5" volumeName="kubernetes.io/configmap/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-daemon-config" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096144 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7991c77-7106-4c73-8812-ff6d4c06b666" volumeName="kubernetes.io/projected/a7991c77-7106-4c73-8812-ff6d4c06b666-kube-api-access-b7mxs" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096155 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8b2b155-18ef-40a1-b87b-a6e44e130d65" volumeName="kubernetes.io/projected/a8b2b155-18ef-40a1-b87b-a6e44e130d65-kube-api-access-9dxcj" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096168 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2aa236a-4ff4-4b39-90ff-c782721444ee" volumeName="kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-image-registry-private-configuration" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096179 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4093a5e-62e9-4f40-b3ad-15453e4f1bcf" volumeName="kubernetes.io/configmap/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf-signing-cabundle" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096192 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5b0a3f1-5189-4572-9765-20eb3960fbae" volumeName="kubernetes.io/projected/c5b0a3f1-5189-4572-9765-20eb3960fbae-kube-api-access-245fq" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096205 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5dd40be-e515-4fe8-be55-b93f5b97973e" volumeName="kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096217 5995 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="747ed7ed-d05b-434a-8360-acc2292c2257" volumeName="kubernetes.io/secret/747ed7ed-d05b-434a-8360-acc2292c2257-tls-certificates" seLinuxMountContext="" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096228 5995 reconstruct.go:97] "Volume reconstruction finished" Nov 04 01:10:50.096274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.096236 5995 reconciler.go:29] "Reconciler: start to sync state" Nov 04 01:10:50.106725 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.106698 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:10:50.109082 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.109058 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:10:50.119902 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.119844 5995 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 01:10:50.121298 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.121271 5995 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 01:10:50.121298 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.121298 5995 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 01:10:50.121492 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.121320 5995 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 01:10:50.121492 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:10:50.121361 5995 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 04 01:10:50.159527 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.159469 5995 generic.go:356] "Generic (PLEG): container finished" podID="c3d08d00-e371-4793-955e-c5b51eef7879" containerID="1985d867be774208da17fe985b40d74948d4d2c980ca8ede515f7cfbb987339a" exitCode=0 Nov 04 01:10:50.159527 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.159524 5995 generic.go:356] "Generic (PLEG): container finished" podID="c3d08d00-e371-4793-955e-c5b51eef7879" containerID="f22b984b447dcd40a855acc1ef8783116a6005d7e31a639ac5257b4d1bcd5bc7" exitCode=0 Nov 04 01:10:50.159527 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.159534 5995 generic.go:356] "Generic (PLEG): container finished" podID="c3d08d00-e371-4793-955e-c5b51eef7879" containerID="2073dc217017c30a1734dddbfb01109307805494a8f6e42b0a4fa6584709a574" exitCode=0 Nov 04 01:10:50.159753 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.159544 5995 generic.go:356] "Generic (PLEG): container finished" podID="c3d08d00-e371-4793-955e-c5b51eef7879" containerID="69d11914966474f22c8f724cb3564599307f7405d0d239dbdbf2810fd734636d" exitCode=0 Nov 04 01:10:50.159753 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.159552 5995 generic.go:356] "Generic (PLEG): container finished" podID="c3d08d00-e371-4793-955e-c5b51eef7879" containerID="d174cba8a906028140ba5b2f6b494efc339d6f8290cc98b395e007b50c2ef9f1" exitCode=0 Nov 04 01:10:50.159753 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.159558 5995 generic.go:356] "Generic (PLEG): container finished" podID="c3d08d00-e371-4793-955e-c5b51eef7879" containerID="9064fb38fafb124af725ba06fa575206f4950d82090bb2e06815146a9d61fce9" exitCode=0 Nov 04 01:10:50.176525 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.176458 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:10:50.176525 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.176489 5995 generic.go:356] "Generic (PLEG): container finished" podID="ae64edd1-43dc-406d-bcad-f7f34bb1f991" containerID="86f7eda2eef130111d4393a379809aa2517d0821e73d020723ac0d438c164361" exitCode=255 Nov 04 01:10:50.189575 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.189544 5995 generic.go:356] "Generic (PLEG): container finished" podID="825c4932c3373ac03b157d52d106cbb9" containerID="380876e5dd3f8ead4729c25a92baf91b3c52503c3326f032a866851288805b17" exitCode=0 Nov 04 01:10:50.189760 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:10:50.189746 5995 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-179.us-west-2.compute.internal\" not found" Nov 04 01:10:50.219246 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.219223 5995 manager.go:324] Recovery completed Nov 04 01:10:50.221512 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:10:50.221494 5995 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 04 01:10:50.243465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.243430 5995 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 01:10:50.245536 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.245519 5995 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 01:10:50.245635 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.245550 5995 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 01:10:50.245635 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.245564 5995 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 01:10:50.248761 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.248742 5995 cpu_manager.go:226] "Starting" policy="none" Nov 04 01:10:50.248761 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.248757 5995 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 01:10:50.248864 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.248765 5995 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 01:10:50.248864 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.248777 5995 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 01:10:50.248941 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.248924 5995 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 04 01:10:50.248972 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.248939 5995 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 04 01:10:50.248972 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.248960 5995 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 04 01:10:50.248972 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.248966 5995 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 04 01:10:50.248972 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.248971 5995 policy_none.go:50] "Start" Nov 04 01:10:50.249096 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.248981 5995 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 01:10:50.249096 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.248992 5995 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 01:10:50.249892 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.249876 5995 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 01:10:50.249955 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.249903 5995 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 01:10:50.249955 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.249914 5995 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 01:10:50.251926 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.251912 5995 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 04 01:10:50.251972 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.251928 5995 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 04 01:10:50.251972 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.251935 5995 policy_none.go:44] "Start" Nov 04 01:10:50.256642 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.256615 5995 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 01:10:50.259198 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.259185 5995 manager.go:341] "Starting Device Plugin manager" Nov 04 01:10:50.259290 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:10:50.259212 5995 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 01:10:50.259290 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.259223 5995 server.go:87] "Starting device plugin registration server" Nov 04 01:10:50.259556 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.259542 5995 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 01:10:50.259620 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.259558 5995 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 01:10:50.259666 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.259641 5995 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 01:10:50.259901 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.259775 5995 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 01:10:50.259901 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.259829 5995 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 01:10:50.259901 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.259836 5995 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 01:10:50.260456 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.259876 5995 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T01:10:50.259818785Z","UUID":"947906d1-c003-4049-b872-4ca1f03cf24f","Handler":null,"Name":"","Endpoint":""} Nov 04 01:10:50.261381 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:10:50.261362 5995 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 01:10:50.261477 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:10:50.261397 5995 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-179.us-west-2.compute.internal\" not found" Nov 04 01:10:50.261971 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.261950 5995 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 01:10:50.262051 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.261982 5995 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 01:10:50.262973 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.262734 5995 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 01:10:50.263272 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.263254 5995 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 01:10:50.263684 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.263650 5995 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 01:10:50.263904 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.263887 5995 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 04 01:10:50.310126 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.310102 5995 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 01:10:50.359818 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.359796 5995 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 01:10:50.360943 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.360923 5995 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 01:10:50.361035 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.360960 5995 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 01:10:50.361035 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.360974 5995 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-179.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 01:10:50.361035 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.361021 5995 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:50.422400 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.422363 5995 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal"] Nov 04 01:10:50.423042 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.423004 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" event={"ID":"5be03f35a536944b4cf2058a360cdcb6","Type":"ContainerStarted","Data":"a5b67596bec2328d8975cc1759f5de4cba7538ae734f75497915ec10209e222d"} Nov 04 01:10:50.423042 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.423032 5995 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:50.423183 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.423047 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" event={"ID":"5be03f35a536944b4cf2058a360cdcb6","Type":"ContainerStarted","Data":"12b8e2e7d0b1cfe4c92e5f13d3fe78740de4021fc7f21746a5bee63bba5a4ec4"} Nov 04 01:10:50.423183 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.423140 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" event={"ID":"825c4932c3373ac03b157d52d106cbb9","Type":"ContainerStarted","Data":"1fd3854760d63fc9afe23eb5fd429ff393881de388059fd28eead474b7545665"} Nov 04 01:10:50.423183 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.423154 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" event={"ID":"825c4932c3373ac03b157d52d106cbb9","Type":"ContainerDied","Data":"380876e5dd3f8ead4729c25a92baf91b3c52503c3326f032a866851288805b17"} Nov 04 01:10:50.423183 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.423166 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" event={"ID":"825c4932c3373ac03b157d52d106cbb9","Type":"ContainerStarted","Data":"13d5686dc21a2275c049fb9f51447a8f8a3652ae89ee45e5e95bc5efead89a01"} Nov 04 01:10:50.423183 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.423176 5995 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:50.463799 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.463778 5995 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:50.463873 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.463829 5995 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:50.486849 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.486826 5995 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 01:10:50.486942 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:10:50.486867 5995 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:50.487518 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.487503 5995 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 01:10:50.487564 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:10:50.487537 5995 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:50.564976 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.564950 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5be03f35a536944b4cf2058a360cdcb6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal\" (UID: \"5be03f35a536944b4cf2058a360cdcb6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:50.565064 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.564981 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/825c4932c3373ac03b157d52d106cbb9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal\" (UID: \"825c4932c3373ac03b157d52d106cbb9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:50.565064 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.565007 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/825c4932c3373ac03b157d52d106cbb9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal\" (UID: \"825c4932c3373ac03b157d52d106cbb9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:50.666066 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.666021 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5be03f35a536944b4cf2058a360cdcb6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal\" (UID: \"5be03f35a536944b4cf2058a360cdcb6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:50.666256 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.666067 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5be03f35a536944b4cf2058a360cdcb6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal\" (UID: \"5be03f35a536944b4cf2058a360cdcb6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:50.666256 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.666092 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/825c4932c3373ac03b157d52d106cbb9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal\" (UID: \"825c4932c3373ac03b157d52d106cbb9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:50.666256 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.666125 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/825c4932c3373ac03b157d52d106cbb9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal\" (UID: \"825c4932c3373ac03b157d52d106cbb9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:50.666256 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.666172 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/825c4932c3373ac03b157d52d106cbb9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal\" (UID: \"825c4932c3373ac03b157d52d106cbb9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:50.666256 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:50.666172 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/825c4932c3373ac03b157d52d106cbb9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal\" (UID: \"825c4932c3373ac03b157d52d106cbb9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:51.062546 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.062518 5995 apiserver.go:52] "Watching apiserver" Nov 04 01:10:51.164056 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.164026 5995 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 01:10:51.164789 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.164764 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf","openshift-network-operator/iptables-alerter-rmvz5","openshift-dns/dns-default-2h8lq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k","openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz","openshift-network-console/networking-console-plugin-86c478bb88-582gz","openshift-ovn-kubernetes/ovnkube-node-8dvnf","openshift-dns/node-resolver-xsmlx","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2w74l","kube-system/global-pull-secret-syncer-b2jgm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx","kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal","openshift-service-ca/service-ca-5fffc97bdf-fk7k8","openshift-console/downloads-6dd789c8b7-gsvcf","openshift-insights/insights-operator-7c4986d449-dgdxz","openshift-console-operator/console-operator-7cd56b8dbc-8cfg7","openshift-ingress/router-default-f5c95ffcf-hl6wf","openshift-cluster-node-tuning-operator/tuned-vdmn8","kube-system/konnectivity-agent-qvjtd","openshift-kube-storage-version-migrator/migrator-66df5d496b-vkk6b","openshift-multus/multus-zn87s","openshift-network-diagnostics/network-check-target-xqtpx","kube-system/kas-connection-checker-685895c7c9-65nzk","kube-system/kas-connection-checker-685895c7c9-xzhqb","openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl","openshift-image-registry/image-registry-69bf8d55f8-x6rb7","openshift-image-registry/node-ca-zzx84","openshift-insights/periodic-gathering-jmkw9-6xlv6","openshift-image-registry/image-registry-6556fd5cf6-2dl7j","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-47hlt","openshift-ingress-canary/ingress-canary-ms2pt","openshift-multus/multus-additional-cni-plugins-dg8wm","openshift-network-diagnostics/network-check-source-6f78d59f99-dbnq4","openshift-kube-storage-version-migrator/migrator-66df5d496b-db6x7","kube-system/kas-connection-checker-685895c7c9-d26rm","openshift-insights/insights-runtime-extractor-4j876","openshift-multus/network-metrics-daemon-zfj7b"] Nov 04 01:10:51.191750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.191724 5995 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 01:10:51.201994 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.201974 5995 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:51.202129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.202116 5995 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:51.214480 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.214462 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.214626 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.214608 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.214711 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.214698 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.215130 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.215114 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 04 01:10:51.215201 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.215174 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.215201 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.215193 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.215278 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.215193 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.215278 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.215228 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-qmn7f\"" Nov 04 01:10:51.215278 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.215253 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 04 01:10:51.215382 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.215318 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ngfvk\"" Nov 04 01:10:51.219938 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.219915 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.220099 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.220078 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9qr5d\"" Nov 04 01:10:51.220179 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.220130 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 04 01:10:51.220179 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.220134 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ltqgp\"" Nov 04 01:10:51.220278 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.220205 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 04 01:10:51.220278 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.220263 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 04 01:10:51.220370 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.220267 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 04 01:10:51.269274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.269250 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cecaa47-44ef-440e-80eb-45230fe3edc0-tmp\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:51.269386 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.269282 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-bound-sa-token\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:51.269386 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.269311 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-systemd\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.269386 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.269330 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-host\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.269386 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.269357 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-kubelet\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.269386 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.269382 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b44a2a16-b00a-4113-b0ab-5cdcb955b325-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-29mpf\" (UID: \"b44a2a16-b00a-4113-b0ab-5cdcb955b325\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" Nov 04 01:10:51.269638 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.269392 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cecaa47-44ef-440e-80eb-45230fe3edc0-tmp\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:51.269638 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.269469 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-cni-binary-copy\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.269638 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.269521 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7573b3c8-fd6e-41c5-9de4-9c441f2ce026-serving-cert\") pod \"service-ca-operator-544d749c8c-d2dsz\" (UID: \"7573b3c8-fd6e-41c5-9de4-9c441f2ce026\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" Nov 04 01:10:51.269638 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.269557 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dxcj\" (UniqueName: \"kubernetes.io/projected/a8b2b155-18ef-40a1-b87b-a6e44e130d65-kube-api-access-9dxcj\") pod \"kas-connection-checker-685895c7c9-xzhqb\" (UID: \"a8b2b155-18ef-40a1-b87b-a6e44e130d65\") " pod="kube-system/kas-connection-checker-685895c7c9-xzhqb" Nov 04 01:10:51.269638 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.269613 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p78dk\" (UniqueName: \"kubernetes.io/projected/6250a398-c824-4c6e-9083-c7b0df543700-kube-api-access-p78dk\") pod \"network-check-source-6f78d59f99-dbnq4\" (UID: \"6250a398-c824-4c6e-9083-c7b0df543700\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dbnq4" Nov 04 01:10:51.270009 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.269981 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ae06262-6e59-4775-bedc-5305ace1fae6-dbus\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:51.270085 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270068 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlvzh\" (UniqueName: \"kubernetes.io/projected/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-kube-api-access-xlvzh\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:51.270219 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270161 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/75c5b017-8f43-41dc-b79b-0353de290289-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:51.270341 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270260 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae64edd1-43dc-406d-bcad-f7f34bb1f991-serving-cert\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:51.270341 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270284 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7573b3c8-fd6e-41c5-9de4-9c441f2ce026-serving-cert\") pod \"service-ca-operator-544d749c8c-d2dsz\" (UID: \"7573b3c8-fd6e-41c5-9de4-9c441f2ce026\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" Nov 04 01:10:51.270341 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270297 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl9gz\" (UniqueName: \"kubernetes.io/projected/f2df8dc5-6403-4827-b6d6-9c209c595e05-kube-api-access-jl9gz\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:51.270341 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270321 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-hostroot\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.270547 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270352 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-image-registry-private-configuration\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:51.270547 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270382 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ae06262-6e59-4775-bedc-5305ace1fae6-kubelet-config\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:51.270547 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270437 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxsvl\" (UniqueName: \"kubernetes.io/projected/8ddedc2c-735e-41e5-a5a9-3386ed890ee1-kube-api-access-kxsvl\") pod \"migrator-66df5d496b-vkk6b\" (UID: \"8ddedc2c-735e-41e5-a5a9-3386ed890ee1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vkk6b" Nov 04 01:10:51.270732 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270626 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-image-registry-private-configuration\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:51.270732 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270686 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-etc-openvswitch\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.270732 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270722 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v22h7\" (UniqueName: \"kubernetes.io/projected/ae64edd1-43dc-406d-bcad-f7f34bb1f991-kube-api-access-v22h7\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:51.270907 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270781 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-etc-kubernetes\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.270907 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270820 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-trusted-ca\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:51.270907 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270856 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7573b3c8-fd6e-41c5-9de4-9c441f2ce026-config\") pod \"service-ca-operator-544d749c8c-d2dsz\" (UID: \"7573b3c8-fd6e-41c5-9de4-9c441f2ce026\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" Nov 04 01:10:51.270907 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270890 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a06edce6-baf3-40bc-9d0e-a6e71c393165-agent-certs\") pod \"konnectivity-agent-qvjtd\" (UID: \"a06edce6-baf3-40bc-9d0e-a6e71c393165\") " pod="kube-system/konnectivity-agent-qvjtd" Nov 04 01:10:51.271105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270926 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-service-ca-bundle\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:51.271105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270955 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-run-systemd\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.271105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.270988 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5cecaa47-44ef-440e-80eb-45230fe3edc0-snapshots\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:51.271105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.271022 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d94r5\" (UniqueName: \"kubernetes.io/projected/5cecaa47-44ef-440e-80eb-45230fe3edc0-kube-api-access-d94r5\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:51.271105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.271058 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3d08d00-e371-4793-955e-c5b51eef7879-system-cni-dir\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:10:51.271105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.271090 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpmgk\" (UniqueName: \"kubernetes.io/projected/82966365-6bec-467d-9279-12ac45dd01e2-kube-api-access-fpmgk\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.271371 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.271116 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-run-ovn\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.271431 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.271399 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-ovn-node-metrics-cert\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.271496 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.271459 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3d08d00-e371-4793-955e-c5b51eef7879-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:10:51.271550 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.271533 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltmvj\" (UniqueName: \"kubernetes.io/projected/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-kube-api-access-ltmvj\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.271623 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.271567 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx7dg\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-kube-api-access-cx7dg\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:51.271623 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.271601 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:51.271711 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.271629 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:51.271711 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.271635 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5cecaa47-44ef-440e-80eb-45230fe3edc0-snapshots\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:51.271711 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.271664 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:51.271711 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.271698 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-ca-trust-extracted\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:51.271875 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.271822 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-582gz\" (UID: \"5b8947a9-b2f0-4de2-a1b8-58342b6b3240\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" Nov 04 01:10:51.271875 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.271836 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7573b3c8-fd6e-41c5-9de4-9c441f2ce026-config\") pod \"service-ca-operator-544d749c8c-d2dsz\" (UID: \"7573b3c8-fd6e-41c5-9de4-9c441f2ce026\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" Nov 04 01:10:51.271875 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.271857 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f50bc9b-91da-4d92-8553-c9f9476665e4-serviceca\") pod \"node-ca-zzx84\" (UID: \"0f50bc9b-91da-4d92-8553-c9f9476665e4\") " pod="openshift-image-registry/node-ca-zzx84" Nov 04 01:10:51.272002 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.271892 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf-signing-key\") pod \"service-ca-5fffc97bdf-fk7k8\" (UID: \"c4093a5e-62e9-4f40-b3ad-15453e4f1bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" Nov 04 01:10:51.272002 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.271951 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5dd40be-e515-4fe8-be55-b93f5b97973e-crio-socket\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:51.272084 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.272049 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a06edce6-baf3-40bc-9d0e-a6e71c393165-konnectivity-ca\") pod \"konnectivity-agent-qvjtd\" (UID: \"a06edce6-baf3-40bc-9d0e-a6e71c393165\") " pod="kube-system/konnectivity-agent-qvjtd" Nov 04 01:10:51.272129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.272081 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-sysconfig\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.272129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.272114 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-sysctl-conf\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.272254 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.272146 5995 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 01:10:51.272254 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:10:51.272198 5995 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:51.272254 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.272214 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03f3058b-0663-48a7-ab67-a41bfe5a2c57-tmp\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.272254 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.272146 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03f3058b-0663-48a7-ab67-a41bfe5a2c57-tmp\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.272409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.272150 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-ca-trust-extracted\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:51.272660 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.272630 5995 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 01:10:51.272775 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:10:51.272684 5995 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" Nov 04 01:10:51.272903 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.272867 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:51.273098 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273038 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f50bc9b-91da-4d92-8553-c9f9476665e4-host\") pod \"node-ca-zzx84\" (UID: \"0f50bc9b-91da-4d92-8553-c9f9476665e4\") " pod="openshift-image-registry/node-ca-zzx84" Nov 04 01:10:51.273098 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273083 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-ovnkube-script-lib\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.273275 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273151 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slglc\" (UniqueName: \"kubernetes.io/projected/ffb55ed6-b225-4db5-a7fb-66a10868befc-kube-api-access-slglc\") pod \"node-resolver-xsmlx\" (UID: \"ffb55ed6-b225-4db5-a7fb-66a10868befc\") " pod="openshift-dns/node-resolver-xsmlx" Nov 04 01:10:51.273275 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273233 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-cni-dir\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.273275 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273264 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d1de04c-5111-415b-bdcc-20dfa9d0e6f1-tmp\") pod \"downloads-6dd789c8b7-gsvcf\" (UID: \"5d1de04c-5111-415b-bdcc-20dfa9d0e6f1\") " pod="openshift-console/downloads-6dd789c8b7-gsvcf" Nov 04 01:10:51.273400 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273289 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8xv5\" (UniqueName: \"kubernetes.io/projected/c3d08d00-e371-4793-955e-c5b51eef7879-kube-api-access-v8xv5\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:10:51.273400 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273319 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-run-multus-certs\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.273400 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273348 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlf5p\" (UniqueName: \"kubernetes.io/projected/03f3058b-0663-48a7-ab67-a41bfe5a2c57-kube-api-access-rlf5p\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.273400 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273379 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-installation-pull-secrets\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:51.273593 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273424 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pbbl\" (UniqueName: \"kubernetes.io/projected/b44a2a16-b00a-4113-b0ab-5cdcb955b325-kube-api-access-2pbbl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-29mpf\" (UID: \"b44a2a16-b00a-4113-b0ab-5cdcb955b325\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" Nov 04 01:10:51.273593 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273453 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-cni-bin\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.273593 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273482 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-env-overrides\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.273593 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273511 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tf5p\" (UniqueName: \"kubernetes.io/projected/cd605a92-f88c-40ea-bde7-53c2da66a576-kube-api-access-9tf5p\") pod \"kas-connection-checker-685895c7c9-d26rm\" (UID: \"cd605a92-f88c-40ea-bde7-53c2da66a576\") " pod="kube-system/kas-connection-checker-685895c7c9-d26rm" Nov 04 01:10:51.273593 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273575 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b44a2a16-b00a-4113-b0ab-5cdcb955b325-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-29mpf\" (UID: \"b44a2a16-b00a-4113-b0ab-5cdcb955b325\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" Nov 04 01:10:51.273802 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273650 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96cde4e-0238-48e6-b343-a0592f6e159e-tmp-dir\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:51.273802 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273680 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:51.273802 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273710 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5b0a3f1-5189-4572-9765-20eb3960fbae-host-slash\") pod \"iptables-alerter-rmvz5\" (UID: \"c5b0a3f1-5189-4572-9765-20eb3960fbae\") " pod="openshift-network-operator/iptables-alerter-rmvz5" Nov 04 01:10:51.273802 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273741 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpfww\" (UniqueName: \"kubernetes.io/projected/a23c835a-aacb-4f92-a2cd-799f5e870b85-kube-api-access-kpfww\") pod \"kas-connection-checker-685895c7c9-65nzk\" (UID: \"a23c835a-aacb-4f92-a2cd-799f5e870b85\") " pod="kube-system/kas-connection-checker-685895c7c9-65nzk" Nov 04 01:10:51.273802 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273795 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3d08d00-e371-4793-955e-c5b51eef7879-cnibin\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:10:51.274012 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273819 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3d08d00-e371-4793-955e-c5b51eef7879-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:10:51.274012 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273849 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.274012 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273888 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-sys-fs\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.274012 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273914 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-slash\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.274012 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273941 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-cni-netd\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.274012 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273966 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae64edd1-43dc-406d-bcad-f7f34bb1f991-config\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:51.274012 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.273995 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-default-certificate\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:51.274338 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274026 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3d08d00-e371-4793-955e-c5b51eef7879-cni-binary-copy\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:10:51.274338 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274057 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3d08d00-e371-4793-955e-c5b51eef7879-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:10:51.274338 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274085 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb55ed6-b225-4db5-a7fb-66a10868befc-tmp-dir\") pod \"node-resolver-xsmlx\" (UID: \"ffb55ed6-b225-4db5-a7fb-66a10868befc\") " pod="openshift-dns/node-resolver-xsmlx" Nov 04 01:10:51.274338 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274126 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:51.274338 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274153 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-run-k8s-cni-cncf-io\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.274338 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274184 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm8jv\" (UniqueName: \"kubernetes.io/projected/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-kube-api-access-pm8jv\") pod \"network-metrics-daemon-zfj7b\" (UID: \"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8\") " pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:10:51.274338 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274222 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsq8f\" (UniqueName: \"kubernetes.io/projected/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-kube-api-access-gsq8f\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:51.274338 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274267 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-sysctl-d\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.274338 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274297 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-var-lib-kubelet\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.274338 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274325 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-tuned\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.275021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274355 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs\") pod \"network-metrics-daemon-zfj7b\" (UID: \"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8\") " pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:10:51.275021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274384 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-modprobe-d\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.275021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274410 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-run\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.275021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274464 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-trusted-ca\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:51.275021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274509 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-nginx-conf\") pod \"networking-console-plugin-86c478bb88-582gz\" (UID: \"5b8947a9-b2f0-4de2-a1b8-58342b6b3240\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" Nov 04 01:10:51.275021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274543 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:51.275021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274573 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxlfx\" (UniqueName: \"kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx\") pod \"network-check-target-xqtpx\" (UID: \"42f02121-70f2-43d1-be9d-2a81425ec97d\") " pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:10:51.275021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274606 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-run-netns\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.275021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274633 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-var-lib-cni-bin\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.275021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274681 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:51.275021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274708 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-registry-tls\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:51.275021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274740 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b5fm\" (UniqueName: \"kubernetes.io/projected/c60eb5db-ec53-40bb-9a72-c712cec7c32c-kube-api-access-4b5fm\") pod \"migrator-66df5d496b-db6x7\" (UID: \"c60eb5db-ec53-40bb-9a72-c712cec7c32c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db6x7" Nov 04 01:10:51.275021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274773 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-ovnkube-config\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.275021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274803 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm4k6\" (UniqueName: \"kubernetes.io/projected/75c5b017-8f43-41dc-b79b-0353de290289-kube-api-access-dm4k6\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:51.275021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274835 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae64edd1-43dc-406d-bcad-f7f34bb1f991-trusted-ca\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:51.275021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274880 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks2wj\" (UniqueName: \"kubernetes.io/projected/e96cde4e-0238-48e6-b343-a0592f6e159e-kube-api-access-ks2wj\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:51.275021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274916 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cecaa47-44ef-440e-80eb-45230fe3edc0-serving-cert\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:51.275021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.274975 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:51.276976 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.275031 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d1de04c-5111-415b-bdcc-20dfa9d0e6f1-tmp\") pod \"downloads-6dd789c8b7-gsvcf\" (UID: \"5d1de04c-5111-415b-bdcc-20dfa9d0e6f1\") " pod="openshift-console/downloads-6dd789c8b7-gsvcf" Nov 04 01:10:51.276976 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.275206 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb55ed6-b225-4db5-a7fb-66a10868befc-tmp-dir\") pod \"node-resolver-xsmlx\" (UID: \"ffb55ed6-b225-4db5-a7fb-66a10868befc\") " pod="openshift-dns/node-resolver-xsmlx" Nov 04 01:10:51.276976 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.275906 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:51.276976 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.276299 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-tuned\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.276976 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.276283 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e96cde4e-0238-48e6-b343-a0592f6e159e-tmp-dir\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:51.276976 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.276732 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:51.276976 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.276777 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tgzg\" (UniqueName: \"kubernetes.io/projected/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-api-access-2tgzg\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:51.276976 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.276831 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7mxs\" (UniqueName: \"kubernetes.io/projected/a7991c77-7106-4c73-8812-ff6d4c06b666-kube-api-access-b7mxs\") pod \"volume-data-source-validator-58c9f5f46d-2w74l\" (UID: \"a7991c77-7106-4c73-8812-ff6d4c06b666\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2w74l" Nov 04 01:10:51.276976 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.276943 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp5d9\" (UniqueName: \"kubernetes.io/projected/7573b3c8-fd6e-41c5-9de4-9c441f2ce026-kube-api-access-dp5d9\") pod \"service-ca-operator-544d749c8c-d2dsz\" (UID: \"7573b3c8-fd6e-41c5-9de4-9c441f2ce026\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" Nov 04 01:10:51.277348 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.276933 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5b017-8f43-41dc-b79b-0353de290289-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:51.277348 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.276975 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-kubernetes\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.277348 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.276976 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b44a2a16-b00a-4113-b0ab-5cdcb955b325-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-29mpf\" (UID: \"b44a2a16-b00a-4113-b0ab-5cdcb955b325\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" Nov 04 01:10:51.277499 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.277440 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-var-lib-openvswitch\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.277564 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.277546 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ffb55ed6-b225-4db5-a7fb-66a10868befc-hosts-file\") pod \"node-resolver-xsmlx\" (UID: \"ffb55ed6-b225-4db5-a7fb-66a10868befc\") " pod="openshift-dns/node-resolver-xsmlx" Nov 04 01:10:51.277608 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.277579 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-systemd-units\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.277655 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.277605 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-socket-dir-parent\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.277655 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.277647 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-sys\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.277736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.277678 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-archives-path\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:51.277736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.277705 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk265\" (UniqueName: \"kubernetes.io/projected/ca77e439-49cb-4ef0-9951-9564ab28ed3e-kube-api-access-lk265\") pod \"ingress-canary-ms2pt\" (UID: \"ca77e439-49cb-4ef0-9951-9564ab28ed3e\") " pod="openshift-ingress-canary/ingress-canary-ms2pt" Nov 04 01:10:51.277736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.277724 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:51.277869 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.277749 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk7ck\" (UniqueName: \"kubernetes.io/projected/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-kube-api-access-bk7ck\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.277869 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.277776 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.277869 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.277806 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-archives-path\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:51.278046 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.278025 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-device-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.278144 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.278122 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-node-log\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.278275 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.278174 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-stats-auth\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:51.278275 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.278233 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-certificates\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:51.278370 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.278343 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-245fq\" (UniqueName: \"kubernetes.io/projected/c5b0a3f1-5189-4572-9765-20eb3960fbae-kube-api-access-245fq\") pod \"iptables-alerter-rmvz5\" (UID: \"c5b0a3f1-5189-4572-9765-20eb3960fbae\") " pod="openshift-network-operator/iptables-alerter-rmvz5" Nov 04 01:10:51.278425 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.278395 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-lib-modules\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.278487 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.278473 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chq5x\" (UniqueName: \"kubernetes.io/projected/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf-kube-api-access-chq5x\") pod \"service-ca-5fffc97bdf-fk7k8\" (UID: \"c4093a5e-62e9-4f40-b3ad-15453e4f1bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" Nov 04 01:10:51.278541 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.278527 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-run-netns\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.278642 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.278629 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-run-openvswitch\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.278685 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.278661 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cecaa47-44ef-440e-80eb-45230fe3edc0-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:51.278745 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.278726 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-system-cni-dir\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.278784 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.278774 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-installation-pull-secrets\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:51.278830 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.278813 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-registry-certificates\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:51.278904 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.278889 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf-signing-cabundle\") pod \"service-ca-5fffc97bdf-fk7k8\" (UID: \"c4093a5e-62e9-4f40-b3ad-15453e4f1bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" Nov 04 01:10:51.278956 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.278916 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-log-socket\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.279029 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279013 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-cnibin\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.279077 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279060 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-827hl\" (UniqueName: \"kubernetes.io/projected/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-kube-api-access-827hl\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:51.279237 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279223 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.279281 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279266 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-var-lib-kubelet\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.279320 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279303 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5b0a3f1-5189-4572-9765-20eb3960fbae-iptables-alerter-script\") pod \"iptables-alerter-rmvz5\" (UID: \"c5b0a3f1-5189-4572-9765-20eb3960fbae\") " pod="openshift-network-operator/iptables-alerter-rmvz5" Nov 04 01:10:51.279350 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279329 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-var-lib-cni-multus\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.279390 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279354 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-daemon-config\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.279438 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279388 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5dd40be-e515-4fe8-be55-b93f5b97973e-data-volume\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:51.279491 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279466 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-socket-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.279526 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279487 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5dd40be-e515-4fe8-be55-b93f5b97973e-data-volume\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:51.279526 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279509 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-trusted-ca-bundle\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:51.279620 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279544 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/747ed7ed-d05b-434a-8360-acc2292c2257-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-47hlt\" (UID: \"747ed7ed-d05b-434a-8360-acc2292c2257\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-47hlt" Nov 04 01:10:51.279620 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279594 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-run-ovn-kubernetes\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.279679 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279624 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-conf-dir\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.279679 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279643 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2aa236a-4ff4-4b39-90ff-c782721444ee-ca-trust-extracted\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:51.279747 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279684 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e96cde4e-0238-48e6-b343-a0592f6e159e-config-volume\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:51.279747 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279710 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2aa236a-4ff4-4b39-90ff-c782721444ee-ca-trust-extracted\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:51.279747 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279716 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca77e439-49cb-4ef0-9951-9564ab28ed3e-cert\") pod \"ingress-canary-ms2pt\" (UID: \"ca77e439-49cb-4ef0-9951-9564ab28ed3e\") " pod="openshift-ingress-canary/ingress-canary-ms2pt" Nov 04 01:10:51.279855 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279787 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rgj\" (UniqueName: \"kubernetes.io/projected/5d1de04c-5111-415b-bdcc-20dfa9d0e6f1-kube-api-access-j8rgj\") pod \"downloads-6dd789c8b7-gsvcf\" (UID: \"5d1de04c-5111-415b-bdcc-20dfa9d0e6f1\") " pod="openshift-console/downloads-6dd789c8b7-gsvcf" Nov 04 01:10:51.279855 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279820 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cecaa47-44ef-440e-80eb-45230fe3edc0-service-ca-bundle\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:51.279951 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279921 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-registration-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.280000 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279957 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-bound-sa-token\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:51.280000 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.279983 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhhtl\" (UniqueName: \"kubernetes.io/projected/0f50bc9b-91da-4d92-8553-c9f9476665e4-kube-api-access-zhhtl\") pod \"node-ca-zzx84\" (UID: \"0f50bc9b-91da-4d92-8553-c9f9476665e4\") " pod="openshift-image-registry/node-ca-zzx84" Nov 04 01:10:51.314780 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.314723 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.315071 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315046 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.315071 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315061 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 04 01:10:51.315071 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315055 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.315071 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315065 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.315071 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315067 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 04 01:10:51.315071 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315112 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.315595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315280 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.315595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315350 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.315595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315360 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.315595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315366 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.315595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315369 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 01:10:51.315808 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315601 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 01:10:51.315808 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315636 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 01:10:51.315808 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315678 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.315808 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315693 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 01:10:51.315987 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315920 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 01:10:51.315987 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315947 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.315987 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315949 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 04 01:10:51.315987 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.315970 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.316177 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.316110 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.316237 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.316185 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-env-overrides\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.316285 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.316240 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3d08d00-e371-4793-955e-c5b51eef7879-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:10:51.317347 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.317318 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.317347 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.317344 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.317505 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.317356 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.317505 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.317389 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.317505 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.317398 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.317505 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.317401 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 04 01:10:51.317505 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.317429 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.317956 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.317942 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.318071 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.318059 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.318112 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.318091 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.319152 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.319136 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf-signing-cabundle\") pod \"service-ca-5fffc97bdf-fk7k8\" (UID: \"c4093a5e-62e9-4f40-b3ad-15453e4f1bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" Nov 04 01:10:51.320352 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.320324 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e96cde4e-0238-48e6-b343-a0592f6e159e-config-volume\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:51.320352 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.320346 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-daemon-config\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.320453 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.320356 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b44a2a16-b00a-4113-b0ab-5cdcb955b325-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-29mpf\" (UID: \"b44a2a16-b00a-4113-b0ab-5cdcb955b325\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" Nov 04 01:10:51.320691 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.320677 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cecaa47-44ef-440e-80eb-45230fe3edc0-service-ca-bundle\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:51.321657 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.321640 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 01:10:51.321736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.321677 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 01:10:51.321736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.321677 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 01:10:51.321736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.321731 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-service-ca-bundle\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:51.322484 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.322465 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.322552 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.322540 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a06edce6-baf3-40bc-9d0e-a6e71c393165-konnectivity-ca\") pod \"konnectivity-agent-qvjtd\" (UID: \"a06edce6-baf3-40bc-9d0e-a6e71c393165\") " pod="kube-system/konnectivity-agent-qvjtd" Nov 04 01:10:51.322590 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.322555 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3d08d00-e371-4793-955e-c5b51eef7879-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:10:51.324460 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.324437 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.324534 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.324524 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.325345 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.325332 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.325391 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.325382 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.325509 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.325497 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2df8dc5-6403-4827-b6d6-9c209c595e05-service-ca-bundle\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:51.326624 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.326610 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3d08d00-e371-4793-955e-c5b51eef7879-cni-binary-copy\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:10:51.327564 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.327547 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 01:10:51.327822 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.327810 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.328189 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.328177 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 01:10:51.328272 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.328262 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.328306 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.328264 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.328306 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.328277 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 04 01:10:51.330573 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.330555 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-cni-binary-copy\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.330573 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.330569 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca77e439-49cb-4ef0-9951-9564ab28ed3e-cert\") pod \"ingress-canary-ms2pt\" (UID: \"ca77e439-49cb-4ef0-9951-9564ab28ed3e\") " pod="openshift-ingress-canary/ingress-canary-ms2pt" Nov 04 01:10:51.333852 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.333833 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf-signing-key\") pod \"service-ca-5fffc97bdf-fk7k8\" (UID: \"c4093a5e-62e9-4f40-b3ad-15453e4f1bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" Nov 04 01:10:51.337084 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.337068 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:51.359340 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.359317 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.359494 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.359391 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.359494 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.359391 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 04 01:10:51.359494 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.359474 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.359663 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.359495 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 04 01:10:51.359663 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.359541 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 04 01:10:51.359663 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.359556 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 04 01:10:51.359663 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.359645 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 01:10:51.359808 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.359675 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g9sbp\"" Nov 04 01:10:51.359808 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.359682 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 01:10:51.359808 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.359691 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 01:10:51.359808 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.359725 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 01:10:51.359944 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.359812 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 01:10:51.359944 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.359865 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4z2qg\"" Nov 04 01:10:51.360441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.360363 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7mrtt\"" Nov 04 01:10:51.360441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.360383 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-x4nsn\"" Nov 04 01:10:51.360441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.360393 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rjcnb\"" Nov 04 01:10:51.360618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.360454 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 04 01:10:51.360618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.360455 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-8hhs6\"" Nov 04 01:10:51.360618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.360579 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ldpj7\"" Nov 04 01:10:51.360618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.360604 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 04 01:10:51.360765 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.360629 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9dncz\"" Nov 04 01:10:51.360765 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.360639 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 01:10:51.360765 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.360638 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j2s6w\"" Nov 04 01:10:51.360885 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.360786 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qr45v\"" Nov 04 01:10:51.361339 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.361322 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tjkgs\"" Nov 04 01:10:51.361445 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.361399 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-image-registry-private-configuration\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:51.361496 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.361476 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-image-registry-private-configuration\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:51.363016 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.363000 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f50bc9b-91da-4d92-8553-c9f9476665e4-serviceca\") pod \"node-ca-zzx84\" (UID: \"0f50bc9b-91da-4d92-8553-c9f9476665e4\") " pod="openshift-image-registry/node-ca-zzx84" Nov 04 01:10:51.363159 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.363145 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5dd40be-e515-4fe8-be55-b93f5b97973e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:51.363856 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.363841 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-ovnkube-script-lib\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.365241 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.365225 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-default-certificate\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:51.365352 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.365335 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-ovnkube-config\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.366667 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.366648 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-nginx-conf\") pod \"networking-console-plugin-86c478bb88-582gz\" (UID: \"5b8947a9-b2f0-4de2-a1b8-58342b6b3240\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" Nov 04 01:10:51.366722 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.366653 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae64edd1-43dc-406d-bcad-f7f34bb1f991-config\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:51.368607 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.368588 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-stats-auth\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:51.368678 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.368629 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-certificates\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:51.369256 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.369240 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-registry-certificates\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:51.369556 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.369538 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5b0a3f1-5189-4572-9765-20eb3960fbae-iptables-alerter-script\") pod \"iptables-alerter-rmvz5\" (UID: \"c5b0a3f1-5189-4572-9765-20eb3960fbae\") " pod="openshift-network-operator/iptables-alerter-rmvz5" Nov 04 01:10:51.371142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.371125 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-ltlc5\"" Nov 04 01:10:51.373185 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.373170 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 04 01:10:51.373185 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.373178 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-f6ftn\"" Nov 04 01:10:51.373306 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.373186 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9sllv\"" Nov 04 01:10:51.373306 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.373246 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 04 01:10:51.373306 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.373253 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 04 01:10:51.373306 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.373282 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d8jdw\"" Nov 04 01:10:51.373306 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.373293 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j9t67\"" Nov 04 01:10:51.373306 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.373303 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 04 01:10:51.373654 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.373303 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-4dm6v\"" Nov 04 01:10:51.373654 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.373341 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 04 01:10:51.373654 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.373429 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-d62tf\"" Nov 04 01:10:51.373765 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.373666 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-bound-sa-token\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:51.373839 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.373823 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p78dk\" (UniqueName: \"kubernetes.io/projected/6250a398-c824-4c6e-9083-c7b0df543700-kube-api-access-p78dk\") pod \"network-check-source-6f78d59f99-dbnq4\" (UID: \"6250a398-c824-4c6e-9083-c7b0df543700\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dbnq4" Nov 04 01:10:51.374165 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.374148 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 01:10:51.374251 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.374212 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8lz6w\"" Nov 04 01:10:51.374305 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.374297 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-prqm6\"" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.374367 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.374626 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-55klw\"" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.374676 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hz84q\"" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.374914 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpmgk\" (UniqueName: \"kubernetes.io/projected/82966365-6bec-467d-9279-12ac45dd01e2-kube-api-access-fpmgk\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.374920 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tf5p\" (UniqueName: \"kubernetes.io/projected/cd605a92-f88c-40ea-bde7-53c2da66a576-kube-api-access-9tf5p\") pod \"kas-connection-checker-685895c7c9-d26rm\" (UID: \"cd605a92-f88c-40ea-bde7-53c2da66a576\") " pod="kube-system/kas-connection-checker-685895c7c9-d26rm" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.374933 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks2wj\" (UniqueName: \"kubernetes.io/projected/e96cde4e-0238-48e6-b343-a0592f6e159e-kube-api-access-ks2wj\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.374923 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl9gz\" (UniqueName: \"kubernetes.io/projected/f2df8dc5-6403-4827-b6d6-9c209c595e05-kube-api-access-jl9gz\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.374924 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b5fm\" (UniqueName: \"kubernetes.io/projected/c60eb5db-ec53-40bb-9a72-c712cec7c32c-kube-api-access-4b5fm\") pod \"migrator-66df5d496b-db6x7\" (UID: \"c60eb5db-ec53-40bb-9a72-c712cec7c32c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db6x7" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.374957 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375025 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375043 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375046 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q5jlj\"" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375059 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375075 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v22h7\" (UniqueName: \"kubernetes.io/projected/ae64edd1-43dc-406d-bcad-f7f34bb1f991-kube-api-access-v22h7\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375085 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375101 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375210 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tgzg\" (UniqueName: \"kubernetes.io/projected/c5dd40be-e515-4fe8-be55-b93f5b97973e-kube-api-access-2tgzg\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375390 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d94r5\" (UniqueName: \"kubernetes.io/projected/5cecaa47-44ef-440e-80eb-45230fe3edc0-kube-api-access-d94r5\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375586 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlf5p\" (UniqueName: \"kubernetes.io/projected/03f3058b-0663-48a7-ab67-a41bfe5a2c57-kube-api-access-rlf5p\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375593 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpfww\" (UniqueName: \"kubernetes.io/projected/a23c835a-aacb-4f92-a2cd-799f5e870b85-kube-api-access-kpfww\") pod \"kas-connection-checker-685895c7c9-65nzk\" (UID: \"a23c835a-aacb-4f92-a2cd-799f5e870b85\") " pod="kube-system/kas-connection-checker-685895c7c9-65nzk" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375611 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-245fq\" (UniqueName: \"kubernetes.io/projected/c5b0a3f1-5189-4572-9765-20eb3960fbae-kube-api-access-245fq\") pod \"iptables-alerter-rmvz5\" (UID: \"c5b0a3f1-5189-4572-9765-20eb3960fbae\") " pod="openshift-network-operator/iptables-alerter-rmvz5" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375623 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8xv5\" (UniqueName: \"kubernetes.io/projected/c3d08d00-e371-4793-955e-c5b51eef7879-kube-api-access-v8xv5\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375610 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slglc\" (UniqueName: \"kubernetes.io/projected/ffb55ed6-b225-4db5-a7fb-66a10868befc-kube-api-access-slglc\") pod \"node-resolver-xsmlx\" (UID: \"ffb55ed6-b225-4db5-a7fb-66a10868befc\") " pod="openshift-dns/node-resolver-xsmlx" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375705 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375723 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s46xs\"" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375739 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-installation-pull-secrets\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375759 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-metrics-certs\") pod \"network-metrics-daemon-zfj7b\" (UID: \"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8\") " pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375831 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm4k6\" (UniqueName: \"kubernetes.io/projected/75c5b017-8f43-41dc-b79b-0353de290289-kube-api-access-dm4k6\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375860 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lwspx\"" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.375954 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chq5x\" (UniqueName: \"kubernetes.io/projected/c4093a5e-62e9-4f40-b3ad-15453e4f1bcf-kube-api-access-chq5x\") pod \"service-ca-5fffc97bdf-fk7k8\" (UID: \"c4093a5e-62e9-4f40-b3ad-15453e4f1bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-fk7k8" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.376175 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pbbl\" (UniqueName: \"kubernetes.io/projected/b44a2a16-b00a-4113-b0ab-5cdcb955b325-kube-api-access-2pbbl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-29mpf\" (UID: \"b44a2a16-b00a-4113-b0ab-5cdcb955b325\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.376173 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm8jv\" (UniqueName: \"kubernetes.io/projected/fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8-kube-api-access-pm8jv\") pod \"network-metrics-daemon-zfj7b\" (UID: \"fd2eb46c-efdf-48fc-aaf9-78acb8aaaac8\") " pod="openshift-multus/network-metrics-daemon-zfj7b" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.376225 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.376283 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ggznq\"" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.376294 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx7dg\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-kube-api-access-cx7dg\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.376330 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxsvl\" (UniqueName: \"kubernetes.io/projected/8ddedc2c-735e-41e5-a5a9-3386ed890ee1-kube-api-access-kxsvl\") pod \"migrator-66df5d496b-vkk6b\" (UID: \"8ddedc2c-735e-41e5-a5a9-3386ed890ee1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vkk6b" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.376385 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsq8f\" (UniqueName: \"kubernetes.io/projected/ef7aa61c-2789-49ed-989d-fa77fed0ff0f-kube-api-access-gsq8f\") pod \"cluster-samples-operator-6ddb6c8b94-q449k\" (UID: \"ef7aa61c-2789-49ed-989d-fa77fed0ff0f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q449k" Nov 04 01:10:51.376409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.376385 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7mxs\" (UniqueName: \"kubernetes.io/projected/a7991c77-7106-4c73-8812-ff6d4c06b666-kube-api-access-b7mxs\") pod \"volume-data-source-validator-58c9f5f46d-2w74l\" (UID: \"a7991c77-7106-4c73-8812-ff6d4c06b666\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2w74l" Nov 04 01:10:51.377750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.376466 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltmvj\" (UniqueName: \"kubernetes.io/projected/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-kube-api-access-ltmvj\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.377750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.376535 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae64edd1-43dc-406d-bcad-f7f34bb1f991-trusted-ca\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:51.377750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.376624 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rgj\" (UniqueName: \"kubernetes.io/projected/5d1de04c-5111-415b-bdcc-20dfa9d0e6f1-kube-api-access-j8rgj\") pod \"downloads-6dd789c8b7-gsvcf\" (UID: \"5d1de04c-5111-415b-bdcc-20dfa9d0e6f1\") " pod="openshift-console/downloads-6dd789c8b7-gsvcf" Nov 04 01:10:51.377750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.376750 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cecaa47-44ef-440e-80eb-45230fe3edc0-serving-cert\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:51.377750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.376804 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-registry-tls\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:51.377750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.376817 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-trusted-ca\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:51.377750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.377562 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2df8dc5-6403-4827-b6d6-9c209c595e05-metrics-certs\") pod \"router-default-f5c95ffcf-hl6wf\" (UID: \"f2df8dc5-6403-4827-b6d6-9c209c595e05\") " pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:51.377750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.377587 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:51.378804 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.378789 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ae06262-6e59-4775-bedc-5305ace1fae6-original-pull-secret\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:51.379864 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.379850 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/747ed7ed-d05b-434a-8360-acc2292c2257-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-47hlt\" (UID: \"747ed7ed-d05b-434a-8360-acc2292c2257\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-47hlt" Nov 04 01:10:51.379933 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.379920 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-installation-pull-secrets\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:51.381058 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381037 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/75c5b017-8f43-41dc-b79b-0353de290289-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fh4kl\" (UID: \"75c5b017-8f43-41dc-b79b-0353de290289\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fh4kl" Nov 04 01:10:51.381100 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381091 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae64edd1-43dc-406d-bcad-f7f34bb1f991-serving-cert\") pod \"console-operator-7cd56b8dbc-8cfg7\" (UID: \"ae64edd1-43dc-406d-bcad-f7f34bb1f991\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:51.381164 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381151 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.381200 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381173 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-device-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.381200 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381187 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-node-log\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.381258 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381204 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-lib-modules\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.381258 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381247 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-node-log\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.381326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381256 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.381326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381257 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-device-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.381326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381256 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-run-netns\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.381326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381289 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-run-netns\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.381326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381312 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-run-openvswitch\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.381492 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381335 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-system-cni-dir\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.381492 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381347 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-lib-modules\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.381492 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381358 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-run-openvswitch\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.381492 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381387 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a06edce6-baf3-40bc-9d0e-a6e71c393165-agent-certs\") pod \"konnectivity-agent-qvjtd\" (UID: \"a06edce6-baf3-40bc-9d0e-a6e71c393165\") " pod="kube-system/konnectivity-agent-qvjtd" Nov 04 01:10:51.381492 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381448 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-log-socket\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.381492 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381464 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-system-cni-dir\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.381492 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381489 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-cnibin\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.381681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381504 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-log-socket\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.381681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381519 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.381681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381539 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-cnibin\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.381681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381546 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-var-lib-kubelet\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.381681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381573 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-var-lib-cni-multus\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.381681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381577 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.381681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381598 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-socket-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.381681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381618 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-var-lib-cni-multus\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.381681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381621 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-var-lib-kubelet\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.381681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381625 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-run-ovn-kubernetes\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.381681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381655 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-run-ovn-kubernetes\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.381681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381666 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-socket-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.381681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381679 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-conf-dir\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381716 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-registration-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381755 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-conf-dir\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381758 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-systemd\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381787 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-registration-dir\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381787 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-host\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381820 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-systemd\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381832 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-kubelet\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381851 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-host\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381866 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ae06262-6e59-4775-bedc-5305ace1fae6-dbus\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381904 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-kubelet\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381900 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-hostroot\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381933 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ae06262-6e59-4775-bedc-5305ace1fae6-dbus\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381944 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ae06262-6e59-4775-bedc-5305ace1fae6-kubelet-config\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381963 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-hostroot\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381976 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-etc-openvswitch\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.381991 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ae06262-6e59-4775-bedc-5305ace1fae6-kubelet-config\") pod \"global-pull-secret-syncer-b2jgm\" (UID: \"3ae06262-6e59-4775-bedc-5305ace1fae6\") " pod="kube-system/global-pull-secret-syncer-b2jgm" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382011 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-etc-kubernetes\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382021 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-etc-openvswitch\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382040 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-run-systemd\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382069 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3d08d00-e371-4793-955e-c5b51eef7879-system-cni-dir\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382094 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-run-ovn\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382102 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-run-systemd\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382110 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-trusted-ca\") pod \"image-registry-6556fd5cf6-2dl7j\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382128 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5dd40be-e515-4fe8-be55-b93f5b97973e-crio-socket\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382145 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3d08d00-e371-4793-955e-c5b51eef7879-system-cni-dir\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382151 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-sysconfig\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.382142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382154 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-run-ovn\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382103 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-etc-kubernetes\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382175 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-sysctl-conf\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382179 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5dd40be-e515-4fe8-be55-b93f5b97973e-crio-socket\") pod \"insights-runtime-extractor-4j876\" (UID: \"c5dd40be-e515-4fe8-be55-b93f5b97973e\") " pod="openshift-insights/insights-runtime-extractor-4j876" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382211 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f50bc9b-91da-4d92-8553-c9f9476665e4-host\") pod \"node-ca-zzx84\" (UID: \"0f50bc9b-91da-4d92-8553-c9f9476665e4\") " pod="openshift-image-registry/node-ca-zzx84" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382222 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-sysconfig\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382248 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-cni-dir\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382280 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f50bc9b-91da-4d92-8553-c9f9476665e4-host\") pod \"node-ca-zzx84\" (UID: \"0f50bc9b-91da-4d92-8553-c9f9476665e4\") " pod="openshift-image-registry/node-ca-zzx84" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382291 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-run-multus-certs\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382312 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-cni-bin\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382331 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-cni-dir\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382340 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5b0a3f1-5189-4572-9765-20eb3960fbae-host-slash\") pod \"iptables-alerter-rmvz5\" (UID: \"c5b0a3f1-5189-4572-9765-20eb3960fbae\") " pod="openshift-network-operator/iptables-alerter-rmvz5" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382365 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3d08d00-e371-4793-955e-c5b51eef7879-cnibin\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382373 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5b0a3f1-5189-4572-9765-20eb3960fbae-host-slash\") pod \"iptables-alerter-rmvz5\" (UID: \"c5b0a3f1-5189-4572-9765-20eb3960fbae\") " pod="openshift-network-operator/iptables-alerter-rmvz5" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382333 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-sysctl-conf\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382363 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-run-multus-certs\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382388 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-cni-bin\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382389 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3d08d00-e371-4793-955e-c5b51eef7879-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382407 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3d08d00-e371-4793-955e-c5b51eef7879-cnibin\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382442 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382475 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-sys-fs\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382483 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382505 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-slash\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382524 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3d08d00-e371-4793-955e-c5b51eef7879-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dg8wm\" (UID: \"c3d08d00-e371-4793-955e-c5b51eef7879\") " pod="openshift-multus/multus-additional-cni-plugins-dg8wm" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382526 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82966365-6bec-467d-9279-12ac45dd01e2-sys-fs\") pod \"aws-ebs-csi-driver-node-xlzrx\" (UID: \"82966365-6bec-467d-9279-12ac45dd01e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xlzrx" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382552 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-slash\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382560 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-cni-netd\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382568 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e96cde4e-0238-48e6-b343-a0592f6e159e-metrics-tls\") pod \"dns-default-2h8lq\" (UID: \"e96cde4e-0238-48e6-b343-a0592f6e159e\") " pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382529 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-host-cni-netd\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382602 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-run-k8s-cni-cncf-io\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382620 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-sysctl-d\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382636 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-var-lib-kubelet\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382641 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-run-k8s-cni-cncf-io\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382668 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-modprobe-d\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382689 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-run\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382704 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-var-lib-kubelet\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382709 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-run-netns\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382731 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-run-netns\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382749 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-var-lib-cni-bin\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382760 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-sysctl-d\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382791 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b8947a9-b2f0-4de2-a1b8-58342b6b3240-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-582gz\" (UID: \"5b8947a9-b2f0-4de2-a1b8-58342b6b3240\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-582gz" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382811 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-ovn-node-metrics-cert\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382790 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-run\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382797 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-modprobe-d\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382800 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-kubernetes\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382802 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-host-var-lib-cni-bin\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382839 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-etc-kubernetes\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382851 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-var-lib-openvswitch\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382874 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ffb55ed6-b225-4db5-a7fb-66a10868befc-hosts-file\") pod \"node-resolver-xsmlx\" (UID: \"ffb55ed6-b225-4db5-a7fb-66a10868befc\") " pod="openshift-dns/node-resolver-xsmlx" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382893 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-var-lib-openvswitch\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382900 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-systemd-units\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382915 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-socket-dir-parent\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382939 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-sys\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382954 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ffb55ed6-b225-4db5-a7fb-66a10868befc-hosts-file\") pod \"node-resolver-xsmlx\" (UID: \"ffb55ed6-b225-4db5-a7fb-66a10868befc\") " pod="openshift-dns/node-resolver-xsmlx" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382960 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-systemd-units\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382974 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5-multus-socket-dir-parent\") pod \"multus-zn87s\" (UID: \"8ce3f5ec-19d0-41a4-8e27-454b22fc9ed5\") " pod="openshift-multus/multus-zn87s" Nov 04 01:10:51.383596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.382999 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03f3058b-0663-48a7-ab67-a41bfe5a2c57-sys\") pod \"tuned-vdmn8\" (UID: \"03f3058b-0663-48a7-ab67-a41bfe5a2c57\") " pod="openshift-cluster-node-tuning-operator/tuned-vdmn8" Nov 04 01:10:51.407253 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.406616 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 04 01:10:51.407253 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.406801 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk7ck\" (UniqueName: \"kubernetes.io/projected/89bd09a4-6d44-49ae-9bfa-837e2e5bf915-kube-api-access-bk7ck\") pod \"ovnkube-node-8dvnf\" (UID: \"89bd09a4-6d44-49ae-9bfa-837e2e5bf915\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:51.407253 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.406946 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp5d9\" (UniqueName: \"kubernetes.io/projected/7573b3c8-fd6e-41c5-9de4-9c441f2ce026-kube-api-access-dp5d9\") pod \"service-ca-operator-544d749c8c-d2dsz\" (UID: \"7573b3c8-fd6e-41c5-9de4-9c441f2ce026\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" Nov 04 01:10:51.407253 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.407191 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dxcj\" (UniqueName: \"kubernetes.io/projected/a8b2b155-18ef-40a1-b87b-a6e44e130d65-kube-api-access-9dxcj\") pod \"kas-connection-checker-685895c7c9-xzhqb\" (UID: \"a8b2b155-18ef-40a1-b87b-a6e44e130d65\") " pod="kube-system/kas-connection-checker-685895c7c9-xzhqb" Nov 04 01:10:51.407253 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.407202 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-bound-sa-token\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:51.407674 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.407272 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk265\" (UniqueName: \"kubernetes.io/projected/ca77e439-49cb-4ef0-9951-9564ab28ed3e-kube-api-access-lk265\") pod \"ingress-canary-ms2pt\" (UID: \"ca77e439-49cb-4ef0-9951-9564ab28ed3e\") " pod="openshift-ingress-canary/ingress-canary-ms2pt" Nov 04 01:10:51.407674 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.407574 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhhtl\" (UniqueName: \"kubernetes.io/projected/0f50bc9b-91da-4d92-8553-c9f9476665e4-kube-api-access-zhhtl\") pod \"node-ca-zzx84\" (UID: \"0f50bc9b-91da-4d92-8553-c9f9476665e4\") " pod="openshift-image-registry/node-ca-zzx84" Nov 04 01:10:51.407674 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.407630 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxlfx\" (UniqueName: \"kubernetes.io/projected/42f02121-70f2-43d1-be9d-2a81425ec97d-kube-api-access-sxlfx\") pod \"network-check-target-xqtpx\" (UID: \"42f02121-70f2-43d1-be9d-2a81425ec97d\") " pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:10:51.407834 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.407775 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-827hl\" (UniqueName: \"kubernetes.io/projected/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-kube-api-access-827hl\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:51.407983 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.407964 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlvzh\" (UniqueName: \"kubernetes.io/projected/81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe-kube-api-access-xlvzh\") pod \"image-registry-69bf8d55f8-x6rb7\" (UID: \"81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe\") " pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:51.409968 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.409951 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cecaa47-44ef-440e-80eb-45230fe3edc0-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-dgdxz\" (UID: \"5cecaa47-44ef-440e-80eb-45230fe3edc0\") " pod="openshift-insights/insights-operator-7c4986d449-dgdxz" Nov 04 01:10:51.410153 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.410140 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-trusted-ca-bundle\") pod \"periodic-gathering-jmkw9-6xlv6\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:10:51.469565 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.469536 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:51.487732 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:51.487711 5995 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:10:52.110601 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.110571 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2h8lq" Nov 04 01:10:52.296500 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.296470 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kk42p"] Nov 04 01:10:52.321936 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.321911 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:10:52.322209 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.322187 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.371361 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.371310 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 04 01:10:52.371482 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.371386 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 04 01:10:52.444102 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.444080 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 04 01:10:52.444234 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.444108 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 04 01:10:52.444234 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.444184 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zgtmx\"" Nov 04 01:10:52.474438 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.471608 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:52.492750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.492722 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/820aaae5-2a74-40c6-a009-61edcd7cdfe5-node-exporter-tls\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.492916 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.492762 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/820aaae5-2a74-40c6-a009-61edcd7cdfe5-sys\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.492916 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.492887 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/820aaae5-2a74-40c6-a009-61edcd7cdfe5-node-exporter-wtmp\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.493008 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.492947 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/820aaae5-2a74-40c6-a009-61edcd7cdfe5-node-exporter-textfile\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.493008 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.492997 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc987\" (UniqueName: \"kubernetes.io/projected/820aaae5-2a74-40c6-a009-61edcd7cdfe5-kube-api-access-dc987\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.493074 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.493033 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/820aaae5-2a74-40c6-a009-61edcd7cdfe5-node-exporter-accelerators-collector-config\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.493108 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.493088 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/820aaae5-2a74-40c6-a009-61edcd7cdfe5-root\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.493173 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.493158 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/820aaae5-2a74-40c6-a009-61edcd7cdfe5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.493641 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.493620 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/820aaae5-2a74-40c6-a009-61edcd7cdfe5-metrics-client-ca\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.594885 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.594851 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/820aaae5-2a74-40c6-a009-61edcd7cdfe5-node-exporter-accelerators-collector-config\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.594885 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.594888 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/820aaae5-2a74-40c6-a009-61edcd7cdfe5-root\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.595115 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.594911 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/820aaae5-2a74-40c6-a009-61edcd7cdfe5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.595115 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.594937 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/820aaae5-2a74-40c6-a009-61edcd7cdfe5-metrics-client-ca\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.595115 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.594973 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/820aaae5-2a74-40c6-a009-61edcd7cdfe5-node-exporter-tls\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.595115 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.594982 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/820aaae5-2a74-40c6-a009-61edcd7cdfe5-root\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.595115 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.594998 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/820aaae5-2a74-40c6-a009-61edcd7cdfe5-sys\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.595115 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.595056 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/820aaae5-2a74-40c6-a009-61edcd7cdfe5-node-exporter-wtmp\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.595115 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.595096 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/820aaae5-2a74-40c6-a009-61edcd7cdfe5-node-exporter-textfile\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.595499 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.595131 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc987\" (UniqueName: \"kubernetes.io/projected/820aaae5-2a74-40c6-a009-61edcd7cdfe5-kube-api-access-dc987\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.595499 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.595345 5995 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 01:10:52.595602 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.595527 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/820aaae5-2a74-40c6-a009-61edcd7cdfe5-node-exporter-wtmp\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.595602 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.595548 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/820aaae5-2a74-40c6-a009-61edcd7cdfe5-metrics-client-ca\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.595602 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.595548 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/820aaae5-2a74-40c6-a009-61edcd7cdfe5-node-exporter-accelerators-collector-config\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.595602 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.595592 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/820aaae5-2a74-40c6-a009-61edcd7cdfe5-sys\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.595791 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.595701 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/820aaae5-2a74-40c6-a009-61edcd7cdfe5-node-exporter-textfile\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.597439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.597393 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/820aaae5-2a74-40c6-a009-61edcd7cdfe5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.597564 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.597540 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/820aaae5-2a74-40c6-a009-61edcd7cdfe5-node-exporter-tls\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.653181 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.653122 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc987\" (UniqueName: \"kubernetes.io/projected/820aaae5-2a74-40c6-a009-61edcd7cdfe5-kube-api-access-dc987\") pod \"node-exporter-kk42p\" (UID: \"820aaae5-2a74-40c6-a009-61edcd7cdfe5\") " pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.686012 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.685990 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kk42p" Nov 04 01:10:52.715739 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:52.715710 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820aaae5_2a74_40c6_a009_61edcd7cdfe5.slice/crio-60f0d11e8c1f58c60d9ce83bd317a91181ef0933488445e30575561237204dc9 WatchSource:0}: Error finding container 60f0d11e8c1f58c60d9ce83bd317a91181ef0933488445e30575561237204dc9: Status 404 returned error can't find the container with id 60f0d11e8c1f58c60d9ce83bd317a91181ef0933488445e30575561237204dc9 Nov 04 01:10:52.918788 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:52.918711 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6556fd5cf6-2dl7j"] Nov 04 01:10:53.212652 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:53.212618 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kk42p" event={"ID":"820aaae5-2a74-40c6-a009-61edcd7cdfe5","Type":"ContainerStarted","Data":"60f0d11e8c1f58c60d9ce83bd317a91181ef0933488445e30575561237204dc9"} Nov 04 01:10:53.395326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:53.395296 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:10:54.222279 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:54.222224 5995 generic.go:356] "Generic (PLEG): container finished" podID="5cecaa47-44ef-440e-80eb-45230fe3edc0" containerID="5f8aeb58ea3d2c92298c966553bf407cf70f4b4754e2a66bf668ade4ec92ed53" exitCode=0 Nov 04 01:10:54.222279 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:54.222260 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-dgdxz" event={"ID":"5cecaa47-44ef-440e-80eb-45230fe3edc0","Type":"ContainerDied","Data":"5f8aeb58ea3d2c92298c966553bf407cf70f4b4754e2a66bf668ade4ec92ed53"} Nov 04 01:10:54.222780 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:54.222664 5995 scope.go:122] "RemoveContainer" containerID="5f8aeb58ea3d2c92298c966553bf407cf70f4b4754e2a66bf668ade4ec92ed53" Nov 04 01:10:54.625650 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:54.625007 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-8cfg7" Nov 04 01:10:55.236369 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:55.236337 5995 generic.go:356] "Generic (PLEG): container finished" podID="7573b3c8-fd6e-41c5-9de4-9c441f2ce026" containerID="400a166532130453e6a7e06da907398203e1d01fbbb7a5154b87d9bd35525ed1" exitCode=0 Nov 04 01:10:55.236849 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:55.236439 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" event={"ID":"7573b3c8-fd6e-41c5-9de4-9c441f2ce026","Type":"ContainerDied","Data":"400a166532130453e6a7e06da907398203e1d01fbbb7a5154b87d9bd35525ed1"} Nov 04 01:10:55.236933 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:55.236860 5995 scope.go:122] "RemoveContainer" containerID="400a166532130453e6a7e06da907398203e1d01fbbb7a5154b87d9bd35525ed1" Nov 04 01:10:55.238869 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:55.238845 5995 generic.go:356] "Generic (PLEG): container finished" podID="820aaae5-2a74-40c6-a009-61edcd7cdfe5" containerID="93707455e8eea9705cad1cfbeab89c9d80568ebf13ecadb566a2e4d89535057d" exitCode=0 Nov 04 01:10:55.238958 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:55.238931 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kk42p" event={"ID":"820aaae5-2a74-40c6-a009-61edcd7cdfe5","Type":"ContainerDied","Data":"93707455e8eea9705cad1cfbeab89c9d80568ebf13ecadb566a2e4d89535057d"} Nov 04 01:10:55.240599 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:55.240581 5995 generic.go:356] "Generic (PLEG): container finished" podID="b44a2a16-b00a-4113-b0ab-5cdcb955b325" containerID="e4691df3ebc0757feac0cac105d84bd2f1b27c2a7372a7d0a3db2ba79314143e" exitCode=0 Nov 04 01:10:55.240669 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:55.240632 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" event={"ID":"b44a2a16-b00a-4113-b0ab-5cdcb955b325","Type":"ContainerDied","Data":"e4691df3ebc0757feac0cac105d84bd2f1b27c2a7372a7d0a3db2ba79314143e"} Nov 04 01:10:55.240918 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:55.240904 5995 scope.go:122] "RemoveContainer" containerID="e4691df3ebc0757feac0cac105d84bd2f1b27c2a7372a7d0a3db2ba79314143e" Nov 04 01:10:55.242530 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:55.242514 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-dgdxz" event={"ID":"5cecaa47-44ef-440e-80eb-45230fe3edc0","Type":"ContainerStarted","Data":"3963ba00112cd520df9043ce64dc39601c9b3f28134be708525cac318d9ce25c"} Nov 04 01:10:55.874711 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:55.874684 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-89dsj-g6bj2"] Nov 04 01:10:55.878305 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:55.878281 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-89dsj-g6bj2" Nov 04 01:10:55.924084 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:55.924056 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-archives-path\") pod \"periodic-gathering-89dsj-g6bj2\" (UID: \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\") " pod="openshift-insights/periodic-gathering-89dsj-g6bj2" Nov 04 01:10:55.924228 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:55.924085 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-service-ca-bundle\") pod \"periodic-gathering-89dsj-g6bj2\" (UID: \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\") " pod="openshift-insights/periodic-gathering-89dsj-g6bj2" Nov 04 01:10:55.924228 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:55.924118 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29rlf\" (UniqueName: \"kubernetes.io/projected/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-kube-api-access-29rlf\") pod \"periodic-gathering-89dsj-g6bj2\" (UID: \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\") " pod="openshift-insights/periodic-gathering-89dsj-g6bj2" Nov 04 01:10:55.924228 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:55.924160 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-trusted-ca-bundle\") pod \"periodic-gathering-89dsj-g6bj2\" (UID: \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\") " pod="openshift-insights/periodic-gathering-89dsj-g6bj2" Nov 04 01:10:55.933356 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:55.933330 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-89dsj-g6bj2"] Nov 04 01:10:56.025584 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:56.025543 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-archives-path\") pod \"periodic-gathering-89dsj-g6bj2\" (UID: \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\") " pod="openshift-insights/periodic-gathering-89dsj-g6bj2" Nov 04 01:10:56.025584 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:56.025586 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-service-ca-bundle\") pod \"periodic-gathering-89dsj-g6bj2\" (UID: \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\") " pod="openshift-insights/periodic-gathering-89dsj-g6bj2" Nov 04 01:10:56.025788 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:56.025627 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29rlf\" (UniqueName: \"kubernetes.io/projected/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-kube-api-access-29rlf\") pod \"periodic-gathering-89dsj-g6bj2\" (UID: \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\") " pod="openshift-insights/periodic-gathering-89dsj-g6bj2" Nov 04 01:10:56.025788 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:56.025684 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-trusted-ca-bundle\") pod \"periodic-gathering-89dsj-g6bj2\" (UID: \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\") " pod="openshift-insights/periodic-gathering-89dsj-g6bj2" Nov 04 01:10:56.026015 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:56.025995 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-archives-path\") pod \"periodic-gathering-89dsj-g6bj2\" (UID: \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\") " pod="openshift-insights/periodic-gathering-89dsj-g6bj2" Nov 04 01:10:56.026442 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:56.026399 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-service-ca-bundle\") pod \"periodic-gathering-89dsj-g6bj2\" (UID: \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\") " pod="openshift-insights/periodic-gathering-89dsj-g6bj2" Nov 04 01:10:56.026765 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:56.026744 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-trusted-ca-bundle\") pod \"periodic-gathering-89dsj-g6bj2\" (UID: \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\") " pod="openshift-insights/periodic-gathering-89dsj-g6bj2" Nov 04 01:10:56.084233 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:56.084210 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29rlf\" (UniqueName: \"kubernetes.io/projected/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-kube-api-access-29rlf\") pod \"periodic-gathering-89dsj-g6bj2\" (UID: \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\") " pod="openshift-insights/periodic-gathering-89dsj-g6bj2" Nov 04 01:10:56.193146 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:56.193124 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-89dsj-g6bj2" Nov 04 01:10:56.256595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:56.254648 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-29mpf" event={"ID":"b44a2a16-b00a-4113-b0ab-5cdcb955b325","Type":"ContainerStarted","Data":"1955284d449971ce775576edd4a2cf3517a4b1725ac7da5f7e835a26b8f4d9a3"} Nov 04 01:10:56.267067 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:56.267036 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2dsz" event={"ID":"7573b3c8-fd6e-41c5-9de4-9c441f2ce026","Type":"ContainerStarted","Data":"c983643b1ee2dda3278971ee02efd03f3ba37c9e4181753f67d62df48b356430"} Nov 04 01:10:56.270992 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:56.270968 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kk42p" event={"ID":"820aaae5-2a74-40c6-a009-61edcd7cdfe5","Type":"ContainerStarted","Data":"7acd91099638300e351243a3e152af7bc3f5ea2ce05cbcf7af2c2462c62535f4"} Nov 04 01:10:56.271078 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:56.271017 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kk42p" event={"ID":"820aaae5-2a74-40c6-a009-61edcd7cdfe5","Type":"ContainerStarted","Data":"8a5af243e5a6662c5ff45b60772ce772d8eb0aef8843dc45bed2b8e0fa6b00c9"} Nov 04 01:10:56.565205 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:56.565136 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-89dsj-g6bj2"] Nov 04 01:10:56.573200 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:10:56.573168 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5fee21d_76e6_4355_87e7_5c3c4c3da2b7.slice/crio-a75bb7c919888a3db2e31c7b7fee2c81fdfc113214c315b3d6aef5f683144bea WatchSource:0}: Error finding container a75bb7c919888a3db2e31c7b7fee2c81fdfc113214c315b3d6aef5f683144bea: Status 404 returned error can't find the container with id a75bb7c919888a3db2e31c7b7fee2c81fdfc113214c315b3d6aef5f683144bea Nov 04 01:10:56.594343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:56.593824 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kk42p" podStartSLOduration=2.456637286 podStartE2EDuration="4.593805455s" totalImagesPullingTime="2.061109964s" totalInitContainerRuntime="76.058205ms" isStatefulPod=true podCreationTimestamp="2026-11-04 01:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:56.592642895 +0000 UTC m=+6.624376283" watchObservedRunningTime="2026-11-04 01:10:56.593805455 +0000 UTC m=+6.625538845" Nov 04 01:10:56.891139 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:56.891045 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:10:57.280335 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:57.280295 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-89dsj-g6bj2" event={"ID":"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7","Type":"ContainerStarted","Data":"9740fc8c9628b309665a4b2908a87cfec8ef4154cc59771607643ac0e1c1f344"} Nov 04 01:10:57.280800 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:57.280388 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-89dsj-g6bj2" event={"ID":"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7","Type":"ContainerStarted","Data":"a75bb7c919888a3db2e31c7b7fee2c81fdfc113214c315b3d6aef5f683144bea"} Nov 04 01:10:57.389454 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:57.389377 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-89dsj-g6bj2" podStartSLOduration=2.389357165 podStartE2EDuration="2.389357165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:57.388366998 +0000 UTC m=+7.420100387" watchObservedRunningTime="2026-11-04 01:10:57.389357165 +0000 UTC m=+7.421090556" Nov 04 01:10:57.477087 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:57.477054 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-47hlt" Nov 04 01:10:57.480980 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:57.480945 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-47hlt" Nov 04 01:10:58.102799 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:58.102718 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:58.102959 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:58.102869 5995 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 01:10:58.104326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:58.104297 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-f5c95ffcf-hl6wf" Nov 04 01:10:59.555740 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:10:59.555709 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:11:00.255123 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:00.255087 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:11:00.255279 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:00.255272 5995 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 01:11:01.068700 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:01.068672 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dvnf" Nov 04 01:11:02.541044 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:02.541016 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2h8lq_e96cde4e-0238-48e6-b343-a0592f6e159e/dns/0.log" Nov 04 01:11:02.652880 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:02.652853 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2h8lq_e96cde4e-0238-48e6-b343-a0592f6e159e/kube-rbac-proxy/0.log" Nov 04 01:11:03.218056 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:03.218027 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:11:03.400253 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:03.400225 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69bf8d55f8-x6rb7" Nov 04 01:11:03.957014 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:03.956986 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xsmlx_ffb55ed6-b225-4db5-a7fb-66a10868befc/dns-node-resolver/0.log" Nov 04 01:11:04.156569 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:04.156518 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6556fd5cf6-2dl7j_c2aa236a-4ff4-4b39-90ff-c782721444ee/registry/0.log" Nov 04 01:11:04.359467 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:04.359355 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-69bf8d55f8-x6rb7_81014c6f-b8a1-4f60-abb7-52cfb2c2f9fe/registry/0.log" Nov 04 01:11:04.963353 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:04.963323 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zzx84_0f50bc9b-91da-4d92-8553-c9f9476665e4/node-ca/0.log" Nov 04 01:11:05.157717 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:05.157685 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f5c95ffcf-hl6wf_f2df8dc5-6403-4827-b6d6-9c209c595e05/router/0.log" Nov 04 01:11:05.759026 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:05.758988 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ms2pt_ca77e439-49cb-4ef0-9951-9564ab28ed3e/serve-healthcheck-canary/0.log" Nov 04 01:11:05.958304 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:05.958251 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-q449k_ef7aa61c-2789-49ed-989d-fa77fed0ff0f/cluster-samples-operator/0.log" Nov 04 01:11:06.157286 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:06.157217 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-q449k_ef7aa61c-2789-49ed-989d-fa77fed0ff0f/cluster-samples-operator-watch/0.log" Nov 04 01:11:08.855792 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:08.855741 5995 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvjtd_a06edce6-baf3-40bc-9d0e-a6e71c393165/konnectivity-agent/0.log" limit=1024 Nov 04 01:11:09.268443 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:09.268382 5995 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qvjtd_a06edce6-baf3-40bc-9d0e-a6e71c393165/konnectivity-agent/0.log" limit=1024 Nov 04 01:11:11.350010 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:11.349965 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-gsvcf" event={"ID":"5d1de04c-5111-415b-bdcc-20dfa9d0e6f1","Type":"ContainerStarted","Data":"004b9fb465a603fbc7850d03fced2aa705ea6a649a4a4225da5b2d1eef340881"} Nov 04 01:11:11.350546 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:11.350328 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-gsvcf" Nov 04 01:11:11.365924 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:11.365885 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-gsvcf" Nov 04 01:11:13.153757 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:13.153717 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-jmkw9-6xlv6"] Nov 04 01:11:13.154335 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:13.153950 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" podUID="d2f6e24e-f98d-4b9a-8d51-080c2c38cf29" containerName="insights-gathering" containerID="cri-o://0484d558aa91301cc08167b8a00b236f9f612ae462154a0c1564a0cc797ceb1f" gracePeriod=30 Nov 04 01:11:13.362565 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:13.362530 5995 generic.go:356] "Generic (PLEG): container finished" podID="d2f6e24e-f98d-4b9a-8d51-080c2c38cf29" containerID="0484d558aa91301cc08167b8a00b236f9f612ae462154a0c1564a0cc797ceb1f" exitCode=2 Nov 04 01:11:13.363087 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:13.363066 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" event={"ID":"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29","Type":"ContainerDied","Data":"0484d558aa91301cc08167b8a00b236f9f612ae462154a0c1564a0cc797ceb1f"} Nov 04 01:11:13.496671 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:13.496641 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:11:13.601511 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:13.601478 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-trusted-ca-bundle\") pod \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " Nov 04 01:11:13.601691 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:13.601525 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-service-ca-bundle\") pod \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " Nov 04 01:11:13.601691 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:13.601572 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-archives-path\") pod \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " Nov 04 01:11:13.601691 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:13.601604 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-kube-api-access-827hl\" (UniqueName: \"kubernetes.io/projected/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-kube-api-access-827hl\") pod \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\" (UID: \"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29\") " Nov 04 01:11:13.601891 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:13.601866 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-trusted-ca-bundle" pod "d2f6e24e-f98d-4b9a-8d51-080c2c38cf29" (UID: "d2f6e24e-f98d-4b9a-8d51-080c2c38cf29"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:11:13.601975 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:13.601904 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-archives-path" pod "d2f6e24e-f98d-4b9a-8d51-080c2c38cf29" (UID: "d2f6e24e-f98d-4b9a-8d51-080c2c38cf29"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:11:13.601975 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:13.601926 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-service-ca-bundle" pod "d2f6e24e-f98d-4b9a-8d51-080c2c38cf29" (UID: "d2f6e24e-f98d-4b9a-8d51-080c2c38cf29"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:11:13.604333 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:13.604304 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-kube-api-access-827hl" pod "d2f6e24e-f98d-4b9a-8d51-080c2c38cf29" (UID: "d2f6e24e-f98d-4b9a-8d51-080c2c38cf29"). InnerVolumeSpecName "kube-api-access-827hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:11:13.702734 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:13.702694 5995 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-archives-path\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:11:13.702734 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:13.702728 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-827hl\" (UniqueName: \"kubernetes.io/projected/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-kube-api-access-827hl\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:11:13.702734 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:13.702740 5995 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-trusted-ca-bundle\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:11:13.703182 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:13.702749 5995 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29-service-ca-bundle\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:11:14.369127 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:14.369099 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" Nov 04 01:11:14.369691 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:14.369098 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jmkw9-6xlv6" event={"ID":"d2f6e24e-f98d-4b9a-8d51-080c2c38cf29","Type":"ContainerDied","Data":"83ffd8448ded9b9ec3ef6209f1bf5d4b96993fb173ab544e4321f7611f36e6ce"} Nov 04 01:11:14.369795 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:14.369727 5995 scope.go:122] "RemoveContainer" containerID="0484d558aa91301cc08167b8a00b236f9f612ae462154a0c1564a0cc797ceb1f" Nov 04 01:11:14.525352 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:14.525315 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-jmkw9-6xlv6"] Nov 04 01:11:14.571434 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:14.571378 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-jmkw9-6xlv6"] Nov 04 01:11:16.128401 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:16.128364 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f6e24e-f98d-4b9a-8d51-080c2c38cf29" path="/var/lib/kubelet/pods/d2f6e24e-f98d-4b9a-8d51-080c2c38cf29/volumes" Nov 04 01:11:17.670516 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:17.670468 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qvjtd" Nov 04 01:11:18.233113 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.233073 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" podUID="c2aa236a-4ff4-4b39-90ff-c782721444ee" containerName="registry" containerID="cri-o://e1b61d649bd932d80eca679d54af7f80e16c71675a22d6b846005bd1d901c3ec" gracePeriod=30 Nov 04 01:11:18.391455 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.391396 5995 generic.go:356] "Generic (PLEG): container finished" podID="c2aa236a-4ff4-4b39-90ff-c782721444ee" containerID="e1b61d649bd932d80eca679d54af7f80e16c71675a22d6b846005bd1d901c3ec" exitCode=0 Nov 04 01:11:18.391718 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.391450 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" event={"ID":"c2aa236a-4ff4-4b39-90ff-c782721444ee","Type":"ContainerDied","Data":"e1b61d649bd932d80eca679d54af7f80e16c71675a22d6b846005bd1d901c3ec"} Nov 04 01:11:18.532814 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.532787 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:11:18.647409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.647377 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-kube-api-access-cx7dg\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-kube-api-access-cx7dg\") pod \"c2aa236a-4ff4-4b39-90ff-c782721444ee\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " Nov 04 01:11:18.647585 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.647441 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls\") pod \"c2aa236a-4ff4-4b39-90ff-c782721444ee\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " Nov 04 01:11:18.647585 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.647473 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-installation-pull-secrets\") pod \"c2aa236a-4ff4-4b39-90ff-c782721444ee\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " Nov 04 01:11:18.647585 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.647500 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-certificates\") pod \"c2aa236a-4ff4-4b39-90ff-c782721444ee\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " Nov 04 01:11:18.648171 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.647929 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-certificates" pod "c2aa236a-4ff4-4b39-90ff-c782721444ee" (UID: "c2aa236a-4ff4-4b39-90ff-c782721444ee"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:11:18.648171 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.647935 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2aa236a-4ff4-4b39-90ff-c782721444ee-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2aa236a-4ff4-4b39-90ff-c782721444ee-ca-trust-extracted\") pod \"c2aa236a-4ff4-4b39-90ff-c782721444ee\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " Nov 04 01:11:18.648171 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.648008 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-trusted-ca\") pod \"c2aa236a-4ff4-4b39-90ff-c782721444ee\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " Nov 04 01:11:18.649311 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.648501 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-bound-sa-token\") pod \"c2aa236a-4ff4-4b39-90ff-c782721444ee\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " Nov 04 01:11:18.649311 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.648577 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-image-registry-private-configuration\") pod \"c2aa236a-4ff4-4b39-90ff-c782721444ee\" (UID: \"c2aa236a-4ff4-4b39-90ff-c782721444ee\") " Nov 04 01:11:18.649311 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.648839 5995 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-certificates\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:11:18.649550 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.649347 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-trusted-ca" pod "c2aa236a-4ff4-4b39-90ff-c782721444ee" (UID: "c2aa236a-4ff4-4b39-90ff-c782721444ee"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:11:18.651253 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.651081 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-installation-pull-secrets" pod "c2aa236a-4ff4-4b39-90ff-c782721444ee" (UID: "c2aa236a-4ff4-4b39-90ff-c782721444ee"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:11:18.651253 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.651141 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls" pod "c2aa236a-4ff4-4b39-90ff-c782721444ee" (UID: "c2aa236a-4ff4-4b39-90ff-c782721444ee"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:11:18.651253 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.651217 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-kube-api-access-cx7dg" pod "c2aa236a-4ff4-4b39-90ff-c782721444ee" (UID: "c2aa236a-4ff4-4b39-90ff-c782721444ee"). InnerVolumeSpecName "kube-api-access-cx7dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:11:18.656623 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.651655 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-image-registry-private-configuration" pod "c2aa236a-4ff4-4b39-90ff-c782721444ee" (UID: "c2aa236a-4ff4-4b39-90ff-c782721444ee"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:11:18.656623 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.652792 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-bound-sa-token" pod "c2aa236a-4ff4-4b39-90ff-c782721444ee" (UID: "c2aa236a-4ff4-4b39-90ff-c782721444ee"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:11:18.656623 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.656118 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2aa236a-4ff4-4b39-90ff-c782721444ee-ca-trust-extracted" pod "c2aa236a-4ff4-4b39-90ff-c782721444ee" (UID: "c2aa236a-4ff4-4b39-90ff-c782721444ee"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:11:18.750149 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.750113 5995 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-image-registry-private-configuration\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:11:18.750149 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.750151 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx7dg\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-kube-api-access-cx7dg\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:11:18.750765 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.750169 5995 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-registry-tls\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:11:18.750765 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.750183 5995 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2aa236a-4ff4-4b39-90ff-c782721444ee-installation-pull-secrets\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:11:18.750765 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.750197 5995 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2aa236a-4ff4-4b39-90ff-c782721444ee-ca-trust-extracted\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:11:18.750765 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.750211 5995 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2aa236a-4ff4-4b39-90ff-c782721444ee-trusted-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:11:18.750765 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:18.750224 5995 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2aa236a-4ff4-4b39-90ff-c782721444ee-bound-sa-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:11:19.397872 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:19.397828 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" event={"ID":"c2aa236a-4ff4-4b39-90ff-c782721444ee","Type":"ContainerDied","Data":"44d7b0a2497bef282b17ba2e0e4b41b6cca26c8d3a27b5de2f18c3746254701d"} Nov 04 01:11:19.397872 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:19.397865 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6556fd5cf6-2dl7j" Nov 04 01:11:19.398268 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:19.397917 5995 scope.go:122] "RemoveContainer" containerID="e1b61d649bd932d80eca679d54af7f80e16c71675a22d6b846005bd1d901c3ec" Nov 04 01:11:19.559307 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:19.558741 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xqtpx" Nov 04 01:11:19.598498 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:19.598469 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6556fd5cf6-2dl7j"] Nov 04 01:11:19.602508 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:19.602485 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6556fd5cf6-2dl7j"] Nov 04 01:11:19.641801 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:11:19.641757 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:11:20.140231 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:11:20.140192 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2aa236a-4ff4-4b39-90ff-c782721444ee" path="/var/lib/kubelet/pods/c2aa236a-4ff4-4b39-90ff-c782721444ee/volumes" Nov 04 01:11:49.641000 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:11:49.640960 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:12:19.641130 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:12:19.641095 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:12:32.688631 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:12:32.688597 5995 generic.go:356] "Generic (PLEG): container finished" podID="b5fee21d-76e6-4355-87e7-5c3c4c3da2b7" containerID="9740fc8c9628b309665a4b2908a87cfec8ef4154cc59771607643ac0e1c1f344" exitCode=0 Nov 04 01:12:32.689027 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:12:32.688638 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-89dsj-g6bj2" event={"ID":"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7","Type":"ContainerDied","Data":"9740fc8c9628b309665a4b2908a87cfec8ef4154cc59771607643ac0e1c1f344"} Nov 04 01:12:33.840669 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:12:33.840645 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-89dsj-g6bj2" Nov 04 01:12:33.921907 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:12:33.921878 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-service-ca-bundle\") pod \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\" (UID: \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\") " Nov 04 01:12:33.922036 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:12:33.921915 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-archives-path\") pod \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\" (UID: \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\") " Nov 04 01:12:33.922036 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:12:33.921987 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-trusted-ca-bundle\") pod \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\" (UID: \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\") " Nov 04 01:12:33.922036 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:12:33.922004 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-kube-api-access-29rlf\" (UniqueName: \"kubernetes.io/projected/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-kube-api-access-29rlf\") pod \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\" (UID: \"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7\") " Nov 04 01:12:33.922308 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:12:33.922283 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-service-ca-bundle" pod "b5fee21d-76e6-4355-87e7-5c3c4c3da2b7" (UID: "b5fee21d-76e6-4355-87e7-5c3c4c3da2b7"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:12:33.922367 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:12:33.922325 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-trusted-ca-bundle" pod "b5fee21d-76e6-4355-87e7-5c3c4c3da2b7" (UID: "b5fee21d-76e6-4355-87e7-5c3c4c3da2b7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:12:33.922367 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:12:33.922330 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-archives-path" pod "b5fee21d-76e6-4355-87e7-5c3c4c3da2b7" (UID: "b5fee21d-76e6-4355-87e7-5c3c4c3da2b7"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:12:33.924106 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:12:33.924086 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-kube-api-access-29rlf" pod "b5fee21d-76e6-4355-87e7-5c3c4c3da2b7" (UID: "b5fee21d-76e6-4355-87e7-5c3c4c3da2b7"). InnerVolumeSpecName "kube-api-access-29rlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:12:34.022762 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:12:34.022705 5995 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-trusted-ca-bundle\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:12:34.022762 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:12:34.022728 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29rlf\" (UniqueName: \"kubernetes.io/projected/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-kube-api-access-29rlf\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:12:34.022762 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:12:34.022742 5995 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-service-ca-bundle\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:12:34.022762 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:12:34.022757 5995 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b5fee21d-76e6-4355-87e7-5c3c4c3da2b7-archives-path\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:12:34.702271 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:12:34.702244 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-89dsj-g6bj2" Nov 04 01:12:34.702271 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:12:34.702251 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-89dsj-g6bj2" event={"ID":"b5fee21d-76e6-4355-87e7-5c3c4c3da2b7","Type":"ContainerDied","Data":"a75bb7c919888a3db2e31c7b7fee2c81fdfc113214c315b3d6aef5f683144bea"} Nov 04 01:12:34.702487 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:12:34.702296 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a75bb7c919888a3db2e31c7b7fee2c81fdfc113214c315b3d6aef5f683144bea" Nov 04 01:12:49.639972 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:12:49.639937 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:13:19.640995 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:13:19.640962 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:13:49.641775 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:13:49.641738 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:14:19.641831 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:14:19.641795 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:14:49.643573 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:14:49.643534 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:14:52.151741 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:14:52.151685 5995 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" podUID="5be03f35a536944b4cf2058a360cdcb6" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Nov 04 01:15:19.640251 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:15:19.640210 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:15:49.639759 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:15:49.639717 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:15:50.136837 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:15:50.136803 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:15:50.137760 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:15:50.137739 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:16:19.640570 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:16:19.640533 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:16:49.640527 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:16:49.640487 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:17:19.641371 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:17:19.641287 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:17:49.640891 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:17:49.640847 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:17:50.740600 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.740565 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm"] Nov 04 01:17:50.741111 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.740974 5995 cpu_manager.go:414] "containerMap: removing container" podUID="b5fee21d-76e6-4355-87e7-5c3c4c3da2b7" containerName="insights-gathering" Nov 04 01:17:50.741111 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.740990 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5fee21d-76e6-4355-87e7-5c3c4c3da2b7" containerName="insights-gathering" Nov 04 01:17:50.741111 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.741001 5995 cpu_manager.go:414] "containerMap: removing container" podUID="d2f6e24e-f98d-4b9a-8d51-080c2c38cf29" containerName="insights-gathering" Nov 04 01:17:50.741111 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.741006 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f6e24e-f98d-4b9a-8d51-080c2c38cf29" containerName="insights-gathering" Nov 04 01:17:50.741111 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.741023 5995 cpu_manager.go:414] "containerMap: removing container" podUID="c2aa236a-4ff4-4b39-90ff-c782721444ee" containerName="registry" Nov 04 01:17:50.741111 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.741028 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2aa236a-4ff4-4b39-90ff-c782721444ee" containerName="registry" Nov 04 01:17:50.741111 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.741084 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5fee21d-76e6-4355-87e7-5c3c4c3da2b7" containerName="insights-gathering" Nov 04 01:17:50.741111 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.741093 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f6e24e-f98d-4b9a-8d51-080c2c38cf29" containerName="insights-gathering" Nov 04 01:17:50.741111 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.741101 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2aa236a-4ff4-4b39-90ff-c782721444ee" containerName="registry" Nov 04 01:17:50.743126 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.743111 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.793036 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.793011 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 04 01:17:50.804452 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.804403 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c406e7-7705-4f34-8771-31113b11cd86-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.804662 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.804643 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c406e7-7705-4f34-8771-31113b11cd86-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.804762 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.804748 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c406e7-7705-4f34-8771-31113b11cd86-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.804871 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.804850 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c406e7-7705-4f34-8771-31113b11cd86-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.804990 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.804881 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c406e7-7705-4f34-8771-31113b11cd86-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.804990 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.804902 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c406e7-7705-4f34-8771-31113b11cd86-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.804990 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.804963 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjnrm\" (UniqueName: \"kubernetes.io/projected/72c406e7-7705-4f34-8771-31113b11cd86-kube-api-access-sjnrm\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.805108 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.804996 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c406e7-7705-4f34-8771-31113b11cd86-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.805108 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.805024 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c406e7-7705-4f34-8771-31113b11cd86-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.839358 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.839324 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-zzzbz\"" Nov 04 01:17:50.840302 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.840278 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm"] Nov 04 01:17:50.905906 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.905874 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c406e7-7705-4f34-8771-31113b11cd86-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.906066 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.905911 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c406e7-7705-4f34-8771-31113b11cd86-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.906066 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.905929 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c406e7-7705-4f34-8771-31113b11cd86-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.906066 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.905952 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c406e7-7705-4f34-8771-31113b11cd86-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.906066 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.905973 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c406e7-7705-4f34-8771-31113b11cd86-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.906309 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.906108 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c406e7-7705-4f34-8771-31113b11cd86-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.906309 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.906209 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjnrm\" (UniqueName: \"kubernetes.io/projected/72c406e7-7705-4f34-8771-31113b11cd86-kube-api-access-sjnrm\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.906410 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.906344 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c406e7-7705-4f34-8771-31113b11cd86-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.906480 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.906402 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c406e7-7705-4f34-8771-31113b11cd86-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.906480 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.906463 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c406e7-7705-4f34-8771-31113b11cd86-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.906558 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.906477 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c406e7-7705-4f34-8771-31113b11cd86-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.906558 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.906519 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c406e7-7705-4f34-8771-31113b11cd86-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.906801 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.906780 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c406e7-7705-4f34-8771-31113b11cd86-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.906865 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.906802 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c406e7-7705-4f34-8771-31113b11cd86-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.908507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.908491 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c406e7-7705-4f34-8771-31113b11cd86-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.908739 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.908719 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c406e7-7705-4f34-8771-31113b11cd86-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.965343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.965314 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjnrm\" (UniqueName: \"kubernetes.io/projected/72c406e7-7705-4f34-8771-31113b11cd86-kube-api-access-sjnrm\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:50.965475 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:50.965356 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c406e7-7705-4f34-8771-31113b11cd86-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-qvplm\" (UID: \"72c406e7-7705-4f34-8771-31113b11cd86\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:51.054675 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:51.054598 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:17:51.473523 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:51.473485 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm"] Nov 04 01:17:51.480391 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:17:51.480362 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c406e7_7705_4f34_8771_31113b11cd86.slice/crio-83e9c2f436a7f5a8ee7640e325849967f2c260df3af46592ec8fb97174ceb2bf WatchSource:0}: Error finding container 83e9c2f436a7f5a8ee7640e325849967f2c260df3af46592ec8fb97174ceb2bf: Status 404 returned error can't find the container with id 83e9c2f436a7f5a8ee7640e325849967f2c260df3af46592ec8fb97174ceb2bf Nov 04 01:17:51.482314 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:51.482299 5995 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:17:51.956472 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:51.956441 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" event={"ID":"72c406e7-7705-4f34-8771-31113b11cd86","Type":"ContainerStarted","Data":"83e9c2f436a7f5a8ee7640e325849967f2c260df3af46592ec8fb97174ceb2bf"} Nov 04 01:17:52.678637 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:52.678598 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn"] Nov 04 01:17:52.681875 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:52.681858 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn" Nov 04 01:17:52.730983 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:52.730955 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 04 01:17:52.777378 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:52.777350 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 04 01:17:52.777542 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:52.777384 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-qjp2r\"" Nov 04 01:17:52.778230 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:52.778213 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn"] Nov 04 01:17:52.823010 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:52.822980 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn\" (UID: \"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn" Nov 04 01:17:52.823153 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:52.823027 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4brp\" (UniqueName: \"kubernetes.io/projected/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-kube-api-access-s4brp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn\" (UID: \"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn" Nov 04 01:17:52.823153 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:52.823055 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn\" (UID: \"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn" Nov 04 01:17:52.924322 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:52.924289 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn\" (UID: \"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn" Nov 04 01:17:52.924504 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:52.924349 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn\" (UID: \"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn" Nov 04 01:17:52.924504 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:52.924390 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4brp\" (UniqueName: \"kubernetes.io/projected/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-kube-api-access-s4brp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn\" (UID: \"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn" Nov 04 01:17:52.924699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:52.924681 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn\" (UID: \"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn" Nov 04 01:17:52.924734 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:52.924718 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn\" (UID: \"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn" Nov 04 01:17:52.982584 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:52.982550 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4brp\" (UniqueName: \"kubernetes.io/projected/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-kube-api-access-s4brp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn\" (UID: \"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn" Nov 04 01:17:52.995362 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:52.995340 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn" Nov 04 01:17:53.193131 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:53.193099 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2"] Nov 04 01:17:53.195992 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:53.195964 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" Nov 04 01:17:53.245073 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:53.245007 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-4ldtx\"" Nov 04 01:17:53.252595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:53.252568 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2"] Nov 04 01:17:53.328782 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:53.328751 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6196c46f-afbf-46fa-a523-69fe1682f2ec-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2\" (UID: \"6196c46f-afbf-46fa-a523-69fe1682f2ec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" Nov 04 01:17:53.328942 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:53.328793 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6196c46f-afbf-46fa-a523-69fe1682f2ec-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2\" (UID: \"6196c46f-afbf-46fa-a523-69fe1682f2ec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" Nov 04 01:17:53.328942 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:53.328816 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fcb2\" (UniqueName: \"kubernetes.io/projected/6196c46f-afbf-46fa-a523-69fe1682f2ec-kube-api-access-9fcb2\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2\" (UID: \"6196c46f-afbf-46fa-a523-69fe1682f2ec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" Nov 04 01:17:53.367960 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:53.367928 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn"] Nov 04 01:17:53.374981 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:17:53.374948 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1abf28_0e48_45ab_a58e_c6bd3459e8c7.slice/crio-59368006c2385305a69aca13b79cdcdab361ee1ef5fd4b65e36dd5f95c8c4694 WatchSource:0}: Error finding container 59368006c2385305a69aca13b79cdcdab361ee1ef5fd4b65e36dd5f95c8c4694: Status 404 returned error can't find the container with id 59368006c2385305a69aca13b79cdcdab361ee1ef5fd4b65e36dd5f95c8c4694 Nov 04 01:17:53.429753 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:53.429728 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6196c46f-afbf-46fa-a523-69fe1682f2ec-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2\" (UID: \"6196c46f-afbf-46fa-a523-69fe1682f2ec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" Nov 04 01:17:53.429909 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:53.429757 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6196c46f-afbf-46fa-a523-69fe1682f2ec-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2\" (UID: \"6196c46f-afbf-46fa-a523-69fe1682f2ec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" Nov 04 01:17:53.429909 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:53.429773 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fcb2\" (UniqueName: \"kubernetes.io/projected/6196c46f-afbf-46fa-a523-69fe1682f2ec-kube-api-access-9fcb2\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2\" (UID: \"6196c46f-afbf-46fa-a523-69fe1682f2ec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" Nov 04 01:17:53.430180 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:53.430155 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6196c46f-afbf-46fa-a523-69fe1682f2ec-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2\" (UID: \"6196c46f-afbf-46fa-a523-69fe1682f2ec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" Nov 04 01:17:53.430180 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:53.430172 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6196c46f-afbf-46fa-a523-69fe1682f2ec-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2\" (UID: \"6196c46f-afbf-46fa-a523-69fe1682f2ec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" Nov 04 01:17:53.487942 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:53.487915 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fcb2\" (UniqueName: \"kubernetes.io/projected/6196c46f-afbf-46fa-a523-69fe1682f2ec-kube-api-access-9fcb2\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2\" (UID: \"6196c46f-afbf-46fa-a523-69fe1682f2ec\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" Nov 04 01:17:53.509927 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:53.509869 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" Nov 04 01:17:53.876275 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:53.876194 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2"] Nov 04 01:17:53.883248 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:17:53.883220 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6196c46f_afbf_46fa_a523_69fe1682f2ec.slice/crio-2f65d54a18e291d58d973ba1d2b85ddbc6c75d1fab1bf238167b9ed0c2d11469 WatchSource:0}: Error finding container 2f65d54a18e291d58d973ba1d2b85ddbc6c75d1fab1bf238167b9ed0c2d11469: Status 404 returned error can't find the container with id 2f65d54a18e291d58d973ba1d2b85ddbc6c75d1fab1bf238167b9ed0c2d11469 Nov 04 01:17:53.966443 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:53.966398 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn" event={"ID":"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7","Type":"ContainerStarted","Data":"59368006c2385305a69aca13b79cdcdab361ee1ef5fd4b65e36dd5f95c8c4694"} Nov 04 01:17:53.967494 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:53.967472 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" event={"ID":"6196c46f-afbf-46fa-a523-69fe1682f2ec","Type":"ContainerStarted","Data":"2f65d54a18e291d58d973ba1d2b85ddbc6c75d1fab1bf238167b9ed0c2d11469"} Nov 04 01:17:55.058812 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:55.058774 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f"] Nov 04 01:17:55.062547 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:55.062524 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" Nov 04 01:17:55.145938 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:55.145739 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5600ddae-56c7-4c93-afff-2900207b0819-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f\" (UID: \"5600ddae-56c7-4c93-afff-2900207b0819\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" Nov 04 01:17:55.145938 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:55.145807 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpc6d\" (UniqueName: \"kubernetes.io/projected/5600ddae-56c7-4c93-afff-2900207b0819-kube-api-access-zpc6d\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f\" (UID: \"5600ddae-56c7-4c93-afff-2900207b0819\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" Nov 04 01:17:55.145938 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:55.145834 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5600ddae-56c7-4c93-afff-2900207b0819-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f\" (UID: \"5600ddae-56c7-4c93-afff-2900207b0819\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" Nov 04 01:17:55.161838 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:55.161799 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-4f2k8\"" Nov 04 01:17:55.162897 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:55.162872 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f"] Nov 04 01:17:55.246939 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:55.246896 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5600ddae-56c7-4c93-afff-2900207b0819-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f\" (UID: \"5600ddae-56c7-4c93-afff-2900207b0819\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" Nov 04 01:17:55.247121 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:55.246973 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpc6d\" (UniqueName: \"kubernetes.io/projected/5600ddae-56c7-4c93-afff-2900207b0819-kube-api-access-zpc6d\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f\" (UID: \"5600ddae-56c7-4c93-afff-2900207b0819\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" Nov 04 01:17:55.247121 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:55.247012 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5600ddae-56c7-4c93-afff-2900207b0819-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f\" (UID: \"5600ddae-56c7-4c93-afff-2900207b0819\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" Nov 04 01:17:55.247480 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:55.247453 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5600ddae-56c7-4c93-afff-2900207b0819-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f\" (UID: \"5600ddae-56c7-4c93-afff-2900207b0819\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" Nov 04 01:17:55.247932 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:55.247904 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5600ddae-56c7-4c93-afff-2900207b0819-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f\" (UID: \"5600ddae-56c7-4c93-afff-2900207b0819\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" Nov 04 01:17:55.307111 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:55.307032 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpc6d\" (UniqueName: \"kubernetes.io/projected/5600ddae-56c7-4c93-afff-2900207b0819-kube-api-access-zpc6d\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f\" (UID: \"5600ddae-56c7-4c93-afff-2900207b0819\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" Nov 04 01:17:55.381168 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:55.381131 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" Nov 04 01:17:55.759739 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:55.759698 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f"] Nov 04 01:17:55.773047 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:17:55.773016 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5600ddae_56c7_4c93_afff_2900207b0819.slice/crio-1b36906c849d9516aa0acad273c827d461927cd1df9c9ed1ca63731291084304 WatchSource:0}: Error finding container 1b36906c849d9516aa0acad273c827d461927cd1df9c9ed1ca63731291084304: Status 404 returned error can't find the container with id 1b36906c849d9516aa0acad273c827d461927cd1df9c9ed1ca63731291084304 Nov 04 01:17:55.979135 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:55.979101 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" event={"ID":"5600ddae-56c7-4c93-afff-2900207b0819","Type":"ContainerStarted","Data":"1b36906c849d9516aa0acad273c827d461927cd1df9c9ed1ca63731291084304"} Nov 04 01:17:59.948429 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:59.948380 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn"] Nov 04 01:17:59.954997 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:17:59.954264 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn" Nov 04 01:18:00.053062 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.053027 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-vzcnq\"" Nov 04 01:18:00.056205 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.056178 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn"] Nov 04 01:18:00.056348 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.056236 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb"] Nov 04 01:18:00.060168 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.060147 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc"] Nov 04 01:18:00.060529 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.060487 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb" Nov 04 01:18:00.064195 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.064167 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" Nov 04 01:18:00.092687 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.092663 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn\" (UID: \"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn" Nov 04 01:18:00.092856 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.092711 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn\" (UID: \"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn" Nov 04 01:18:00.092856 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.092834 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpjxv\" (UniqueName: \"kubernetes.io/projected/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-kube-api-access-vpjxv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn\" (UID: \"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn" Nov 04 01:18:00.095238 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.095181 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb"] Nov 04 01:18:00.113854 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.109813 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc"] Nov 04 01:18:00.113854 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.113074 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-4n46n\"" Nov 04 01:18:00.139914 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.139888 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-q4l92\"" Nov 04 01:18:00.193371 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.193330 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpjxv\" (UniqueName: \"kubernetes.io/projected/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-kube-api-access-vpjxv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn\" (UID: \"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn" Nov 04 01:18:00.193371 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.193374 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11af4777-1f84-4d59-a8f0-5abdd866f88d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc\" (UID: \"11af4777-1f84-4d59-a8f0-5abdd866f88d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" Nov 04 01:18:00.193629 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.193430 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f342ad90-5804-4f04-9ca1-991bdec69858-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb\" (UID: \"f342ad90-5804-4f04-9ca1-991bdec69858\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb" Nov 04 01:18:00.193629 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.193459 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f342ad90-5804-4f04-9ca1-991bdec69858-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb\" (UID: \"f342ad90-5804-4f04-9ca1-991bdec69858\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb" Nov 04 01:18:00.193629 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.193512 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn\" (UID: \"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn" Nov 04 01:18:00.193629 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.193553 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11af4777-1f84-4d59-a8f0-5abdd866f88d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc\" (UID: \"11af4777-1f84-4d59-a8f0-5abdd866f88d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" Nov 04 01:18:00.193629 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.193603 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn\" (UID: \"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn" Nov 04 01:18:00.193780 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.193634 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d76nf\" (UniqueName: \"kubernetes.io/projected/11af4777-1f84-4d59-a8f0-5abdd866f88d-kube-api-access-d76nf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc\" (UID: \"11af4777-1f84-4d59-a8f0-5abdd866f88d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" Nov 04 01:18:00.193780 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.193664 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zptj\" (UniqueName: \"kubernetes.io/projected/f342ad90-5804-4f04-9ca1-991bdec69858-kube-api-access-4zptj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb\" (UID: \"f342ad90-5804-4f04-9ca1-991bdec69858\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb" Nov 04 01:18:00.194463 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.194439 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn\" (UID: \"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn" Nov 04 01:18:00.194727 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.194705 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn\" (UID: \"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn" Nov 04 01:18:00.252039 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.251971 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpjxv\" (UniqueName: \"kubernetes.io/projected/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-kube-api-access-vpjxv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn\" (UID: \"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn" Nov 04 01:18:00.275445 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.275224 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn" Nov 04 01:18:00.294432 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.294388 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11af4777-1f84-4d59-a8f0-5abdd866f88d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc\" (UID: \"11af4777-1f84-4d59-a8f0-5abdd866f88d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" Nov 04 01:18:00.294856 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.294836 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f342ad90-5804-4f04-9ca1-991bdec69858-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb\" (UID: \"f342ad90-5804-4f04-9ca1-991bdec69858\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb" Nov 04 01:18:00.294969 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.294887 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f342ad90-5804-4f04-9ca1-991bdec69858-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb\" (UID: \"f342ad90-5804-4f04-9ca1-991bdec69858\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb" Nov 04 01:18:00.294969 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.294943 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11af4777-1f84-4d59-a8f0-5abdd866f88d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc\" (UID: \"11af4777-1f84-4d59-a8f0-5abdd866f88d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" Nov 04 01:18:00.295095 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.294979 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d76nf\" (UniqueName: \"kubernetes.io/projected/11af4777-1f84-4d59-a8f0-5abdd866f88d-kube-api-access-d76nf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc\" (UID: \"11af4777-1f84-4d59-a8f0-5abdd866f88d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" Nov 04 01:18:00.295095 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.295030 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zptj\" (UniqueName: \"kubernetes.io/projected/f342ad90-5804-4f04-9ca1-991bdec69858-kube-api-access-4zptj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb\" (UID: \"f342ad90-5804-4f04-9ca1-991bdec69858\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb" Nov 04 01:18:00.295326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.294774 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11af4777-1f84-4d59-a8f0-5abdd866f88d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc\" (UID: \"11af4777-1f84-4d59-a8f0-5abdd866f88d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" Nov 04 01:18:00.295678 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.295656 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f342ad90-5804-4f04-9ca1-991bdec69858-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb\" (UID: \"f342ad90-5804-4f04-9ca1-991bdec69858\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb" Nov 04 01:18:00.295978 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.295953 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f342ad90-5804-4f04-9ca1-991bdec69858-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb\" (UID: \"f342ad90-5804-4f04-9ca1-991bdec69858\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb" Nov 04 01:18:00.296189 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.296150 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11af4777-1f84-4d59-a8f0-5abdd866f88d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc\" (UID: \"11af4777-1f84-4d59-a8f0-5abdd866f88d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" Nov 04 01:18:00.357836 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.357104 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zptj\" (UniqueName: \"kubernetes.io/projected/f342ad90-5804-4f04-9ca1-991bdec69858-kube-api-access-4zptj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb\" (UID: \"f342ad90-5804-4f04-9ca1-991bdec69858\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb" Nov 04 01:18:00.357836 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.357785 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d76nf\" (UniqueName: \"kubernetes.io/projected/11af4777-1f84-4d59-a8f0-5abdd866f88d-kube-api-access-d76nf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc\" (UID: \"11af4777-1f84-4d59-a8f0-5abdd866f88d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" Nov 04 01:18:00.386305 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.384459 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb" Nov 04 01:18:00.389720 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.389602 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" Nov 04 01:18:00.663759 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.663611 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn"] Nov 04 01:18:00.674628 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:18:00.674595 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f9b4f6_6bcd_4f0f_9c9a_10425c2dfe87.slice/crio-92da3aa3bf980bd4f3f713f7e760032dc509c4f11462b641add987e82c8eb2b6 WatchSource:0}: Error finding container 92da3aa3bf980bd4f3f713f7e760032dc509c4f11462b641add987e82c8eb2b6: Status 404 returned error can't find the container with id 92da3aa3bf980bd4f3f713f7e760032dc509c4f11462b641add987e82c8eb2b6 Nov 04 01:18:00.780766 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.780730 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc"] Nov 04 01:18:00.784251 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.784180 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb"] Nov 04 01:18:00.785326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.785292 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:18:00.785431 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.785377 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:18:00.785497 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:00.785459 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:18:00.788561 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:18:00.788137 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11af4777_1f84_4d59_a8f0_5abdd866f88d.slice/crio-2bb213444aa7a66727c7894986cce5169094b61e4308eabb13da4d91fbdb8286 WatchSource:0}: Error finding container 2bb213444aa7a66727c7894986cce5169094b61e4308eabb13da4d91fbdb8286: Status 404 returned error can't find the container with id 2bb213444aa7a66727c7894986cce5169094b61e4308eabb13da4d91fbdb8286 Nov 04 01:18:00.791050 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:18:00.791024 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf342ad90_5804_4f04_9ca1_991bdec69858.slice/crio-22c72322b56ab0f2d208207f5f84ce3d1fc049a9f503e58bd26f5ffbf1bef91e WatchSource:0}: Error finding container 22c72322b56ab0f2d208207f5f84ce3d1fc049a9f503e58bd26f5ffbf1bef91e: Status 404 returned error can't find the container with id 22c72322b56ab0f2d208207f5f84ce3d1fc049a9f503e58bd26f5ffbf1bef91e Nov 04 01:18:01.008199 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:01.007624 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" event={"ID":"72c406e7-7705-4f34-8771-31113b11cd86","Type":"ContainerStarted","Data":"93d2442e8b63257c530820148d22413af7fb154f3e31f4d7da3ee14783029629"} Nov 04 01:18:01.009733 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:01.009698 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb" event={"ID":"f342ad90-5804-4f04-9ca1-991bdec69858","Type":"ContainerStarted","Data":"22c72322b56ab0f2d208207f5f84ce3d1fc049a9f503e58bd26f5ffbf1bef91e"} Nov 04 01:18:01.011521 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:01.011494 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" event={"ID":"11af4777-1f84-4d59-a8f0-5abdd866f88d","Type":"ContainerStarted","Data":"2bb213444aa7a66727c7894986cce5169094b61e4308eabb13da4d91fbdb8286"} Nov 04 01:18:01.013222 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:01.013200 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn" event={"ID":"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87","Type":"ContainerStarted","Data":"92da3aa3bf980bd4f3f713f7e760032dc509c4f11462b641add987e82c8eb2b6"} Nov 04 01:18:01.055824 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:01.055791 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:18:01.057228 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:01.057194 5995 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Nov 04 01:18:01.057228 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:01.057254 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" podUID="72c406e7-7705-4f34-8771-31113b11cd86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Nov 04 01:18:01.122863 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:01.121624 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" podStartSLOduration=1.819692602 podStartE2EDuration="11.121606126s" totalImagesPullingTime="9.301913524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:01.121118428 +0000 UTC m=+431.152851865" watchObservedRunningTime="2026-11-04 01:18:01.121606126 +0000 UTC m=+431.153339517" Nov 04 01:18:02.059612 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:02.059583 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:18:03.029169 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:03.029074 5995 generic.go:356] "Generic (PLEG): container finished" podID="ec1abf28-0e48-45ab-a58e-c6bd3459e8c7" containerID="1e3f82da816278aa37580be91d5b92894dfcad385c66c855e1f7e2f089dd9c4b" exitCode=0 Nov 04 01:18:03.029338 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:03.029166 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn" event={"ID":"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7","Type":"ContainerDied","Data":"1e3f82da816278aa37580be91d5b92894dfcad385c66c855e1f7e2f089dd9c4b"} Nov 04 01:18:03.034311 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:03.033273 5995 generic.go:356] "Generic (PLEG): container finished" podID="5600ddae-56c7-4c93-afff-2900207b0819" containerID="f73738f52eb24af66c3e38e86eb926380cb88e4394aba09dc06a226597273e07" exitCode=0 Nov 04 01:18:03.034311 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:03.033920 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" event={"ID":"5600ddae-56c7-4c93-afff-2900207b0819","Type":"ContainerDied","Data":"f73738f52eb24af66c3e38e86eb926380cb88e4394aba09dc06a226597273e07"} Nov 04 01:18:03.034311 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:03.034121 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:18:03.035841 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:03.035801 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-qvplm" Nov 04 01:18:04.041296 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:04.041256 5995 generic.go:356] "Generic (PLEG): container finished" podID="6196c46f-afbf-46fa-a523-69fe1682f2ec" containerID="3f013ab424db1368428bbdb8ac5845176e492ef8b0c27aefed92dac0446703a4" exitCode=0 Nov 04 01:18:04.041804 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:04.041481 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" event={"ID":"6196c46f-afbf-46fa-a523-69fe1682f2ec","Type":"ContainerDied","Data":"3f013ab424db1368428bbdb8ac5845176e492ef8b0c27aefed92dac0446703a4"} Nov 04 01:18:06.060641 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:06.058552 5995 generic.go:356] "Generic (PLEG): container finished" podID="6196c46f-afbf-46fa-a523-69fe1682f2ec" containerID="491d4701529f767f7e32eda8ae99e9910b58dfb6a8823a55ce6b08b91bcb5f10" exitCode=0 Nov 04 01:18:06.060641 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:06.058623 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" event={"ID":"6196c46f-afbf-46fa-a523-69fe1682f2ec","Type":"ContainerDied","Data":"491d4701529f767f7e32eda8ae99e9910b58dfb6a8823a55ce6b08b91bcb5f10"} Nov 04 01:18:07.066863 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:07.066787 5995 generic.go:356] "Generic (PLEG): container finished" podID="f342ad90-5804-4f04-9ca1-991bdec69858" containerID="c8942c421f4a15440090ae754dbd4f85d6f21f80cc12c45a38d52b917aa2ed90" exitCode=0 Nov 04 01:18:07.067319 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:07.066922 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb" event={"ID":"f342ad90-5804-4f04-9ca1-991bdec69858","Type":"ContainerDied","Data":"c8942c421f4a15440090ae754dbd4f85d6f21f80cc12c45a38d52b917aa2ed90"} Nov 04 01:18:07.071371 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:07.070832 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" event={"ID":"11af4777-1f84-4d59-a8f0-5abdd866f88d","Type":"ContainerStarted","Data":"677c9ea0c155ef122a959af4fb265e096321ac5d32557e610d4059b3e626c539"} Nov 04 01:18:08.076157 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:08.076121 5995 generic.go:356] "Generic (PLEG): container finished" podID="11af4777-1f84-4d59-a8f0-5abdd866f88d" containerID="677c9ea0c155ef122a959af4fb265e096321ac5d32557e610d4059b3e626c539" exitCode=0 Nov 04 01:18:08.076642 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:08.076213 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" event={"ID":"11af4777-1f84-4d59-a8f0-5abdd866f88d","Type":"ContainerDied","Data":"677c9ea0c155ef122a959af4fb265e096321ac5d32557e610d4059b3e626c539"} Nov 04 01:18:08.077850 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:08.077832 5995 generic.go:356] "Generic (PLEG): container finished" podID="07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87" containerID="14b32d9a976bdbfdda234087a76e0a880337eb2da4938de419f44b61acbf67ca" exitCode=0 Nov 04 01:18:08.077956 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:08.077918 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn" event={"ID":"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87","Type":"ContainerDied","Data":"14b32d9a976bdbfdda234087a76e0a880337eb2da4938de419f44b61acbf67ca"} Nov 04 01:18:08.079950 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:08.079928 5995 generic.go:356] "Generic (PLEG): container finished" podID="ec1abf28-0e48-45ab-a58e-c6bd3459e8c7" containerID="17d6906d4619339ccf74dc1c3f240924b2252f82e81d9f0351632184435f4940" exitCode=0 Nov 04 01:18:08.080039 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:08.079994 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn" event={"ID":"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7","Type":"ContainerDied","Data":"17d6906d4619339ccf74dc1c3f240924b2252f82e81d9f0351632184435f4940"} Nov 04 01:18:08.082020 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:08.081995 5995 generic.go:356] "Generic (PLEG): container finished" podID="5600ddae-56c7-4c93-afff-2900207b0819" containerID="a3080bf4ca96847bce684efc7037c186744e5203e6ee45861bee6674311781bf" exitCode=0 Nov 04 01:18:08.082133 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:08.082076 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" event={"ID":"5600ddae-56c7-4c93-afff-2900207b0819","Type":"ContainerDied","Data":"a3080bf4ca96847bce684efc7037c186744e5203e6ee45861bee6674311781bf"} Nov 04 01:18:09.089185 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:09.089136 5995 generic.go:356] "Generic (PLEG): container finished" podID="f342ad90-5804-4f04-9ca1-991bdec69858" containerID="2cf4db5d87f4c656ba6dcf06365d303e6115c10007337efa2cccba3bcee98425" exitCode=0 Nov 04 01:18:09.089657 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:09.089200 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb" event={"ID":"f342ad90-5804-4f04-9ca1-991bdec69858","Type":"ContainerDied","Data":"2cf4db5d87f4c656ba6dcf06365d303e6115c10007337efa2cccba3bcee98425"} Nov 04 01:18:10.098409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:10.098373 5995 generic.go:356] "Generic (PLEG): container finished" podID="07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87" containerID="d89456419ca373c463283b8d73247f786172fafe3f8b948a87f802d5c69dcaf2" exitCode=0 Nov 04 01:18:10.098875 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:10.098462 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn" event={"ID":"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87","Type":"ContainerDied","Data":"d89456419ca373c463283b8d73247f786172fafe3f8b948a87f802d5c69dcaf2"} Nov 04 01:18:10.101466 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:10.101437 5995 generic.go:356] "Generic (PLEG): container finished" podID="11af4777-1f84-4d59-a8f0-5abdd866f88d" containerID="cb9bfd53739905caffecd84f76660fdc9ae6c145873850f4eb3c03ef7c3161b1" exitCode=0 Nov 04 01:18:10.101559 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:10.101504 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" event={"ID":"11af4777-1f84-4d59-a8f0-5abdd866f88d","Type":"ContainerDied","Data":"cb9bfd53739905caffecd84f76660fdc9ae6c145873850f4eb3c03ef7c3161b1"} Nov 04 01:18:18.165888 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:18.165855 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" event={"ID":"6196c46f-afbf-46fa-a523-69fe1682f2ec","Type":"ContainerStarted","Data":"c3900f7d92afc48990c2bc423285ea00a6d840f50fde2734251085f80df5d860"} Nov 04 01:18:18.279171 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:18.278923 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" podStartSLOduration=3.427395401 podStartE2EDuration="25.278907651s" totalImagesPullingTime="21.85151225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:18.277057655 +0000 UTC m=+448.308791045" watchObservedRunningTime="2026-11-04 01:18:18.278907651 +0000 UTC m=+448.310641040" Nov 04 01:18:19.175775 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:19.175380 5995 generic.go:356] "Generic (PLEG): container finished" podID="11af4777-1f84-4d59-a8f0-5abdd866f88d" containerID="7c00dcfa37953f0341cabbb5780a0d68a9c7619737f9cb16bebb2c853cacc2ab" exitCode=0 Nov 04 01:18:19.175775 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:19.175444 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" event={"ID":"11af4777-1f84-4d59-a8f0-5abdd866f88d","Type":"ContainerDied","Data":"7c00dcfa37953f0341cabbb5780a0d68a9c7619737f9cb16bebb2c853cacc2ab"} Nov 04 01:18:19.178268 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:19.178130 5995 generic.go:356] "Generic (PLEG): container finished" podID="07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87" containerID="f8fa0db05f50e8060a039921c3869bb914df5df88661fcdd7917ba2fef59e820" exitCode=0 Nov 04 01:18:19.178268 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:19.178245 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn" event={"ID":"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87","Type":"ContainerDied","Data":"f8fa0db05f50e8060a039921c3869bb914df5df88661fcdd7917ba2fef59e820"} Nov 04 01:18:19.181486 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:19.181463 5995 generic.go:356] "Generic (PLEG): container finished" podID="6196c46f-afbf-46fa-a523-69fe1682f2ec" containerID="c3900f7d92afc48990c2bc423285ea00a6d840f50fde2734251085f80df5d860" exitCode=0 Nov 04 01:18:19.181581 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:19.181500 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" event={"ID":"6196c46f-afbf-46fa-a523-69fe1682f2ec","Type":"ContainerDied","Data":"c3900f7d92afc48990c2bc423285ea00a6d840f50fde2734251085f80df5d860"} Nov 04 01:18:19.648380 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:18:19.648335 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:18:20.193045 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.193009 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" event={"ID":"5600ddae-56c7-4c93-afff-2900207b0819","Type":"ContainerStarted","Data":"579bed695c48e1e0687f0670a5f7d3f2162507f498924190f716a29be410ad34"} Nov 04 01:18:20.382597 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.382568 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" Nov 04 01:18:20.419054 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.419027 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn" Nov 04 01:18:20.422597 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.422574 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" Nov 04 01:18:20.502857 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.502836 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11af4777-1f84-4d59-a8f0-5abdd866f88d-util\" (UniqueName: \"kubernetes.io/empty-dir/11af4777-1f84-4d59-a8f0-5abdd866f88d-util\") pod \"11af4777-1f84-4d59-a8f0-5abdd866f88d\" (UID: \"11af4777-1f84-4d59-a8f0-5abdd866f88d\") " Nov 04 01:18:20.502961 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.502877 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6196c46f-afbf-46fa-a523-69fe1682f2ec-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6196c46f-afbf-46fa-a523-69fe1682f2ec-bundle\") pod \"6196c46f-afbf-46fa-a523-69fe1682f2ec\" (UID: \"6196c46f-afbf-46fa-a523-69fe1682f2ec\") " Nov 04 01:18:20.502961 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.502905 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6196c46f-afbf-46fa-a523-69fe1682f2ec-util\" (UniqueName: \"kubernetes.io/empty-dir/6196c46f-afbf-46fa-a523-69fe1682f2ec-util\") pod \"6196c46f-afbf-46fa-a523-69fe1682f2ec\" (UID: \"6196c46f-afbf-46fa-a523-69fe1682f2ec\") " Nov 04 01:18:20.502961 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.502939 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6196c46f-afbf-46fa-a523-69fe1682f2ec-kube-api-access-9fcb2\" (UniqueName: \"kubernetes.io/projected/6196c46f-afbf-46fa-a523-69fe1682f2ec-kube-api-access-9fcb2\") pod \"6196c46f-afbf-46fa-a523-69fe1682f2ec\" (UID: \"6196c46f-afbf-46fa-a523-69fe1682f2ec\") " Nov 04 01:18:20.503275 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.502966 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11af4777-1f84-4d59-a8f0-5abdd866f88d-kube-api-access-d76nf\" (UniqueName: \"kubernetes.io/projected/11af4777-1f84-4d59-a8f0-5abdd866f88d-kube-api-access-d76nf\") pod \"11af4777-1f84-4d59-a8f0-5abdd866f88d\" (UID: \"11af4777-1f84-4d59-a8f0-5abdd866f88d\") " Nov 04 01:18:20.503275 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.502998 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-util\" (UniqueName: \"kubernetes.io/empty-dir/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-util\") pod \"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87\" (UID: \"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87\") " Nov 04 01:18:20.503275 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.503014 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11af4777-1f84-4d59-a8f0-5abdd866f88d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/11af4777-1f84-4d59-a8f0-5abdd866f88d-bundle\") pod \"11af4777-1f84-4d59-a8f0-5abdd866f88d\" (UID: \"11af4777-1f84-4d59-a8f0-5abdd866f88d\") " Nov 04 01:18:20.503275 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.503036 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-bundle\" (UniqueName: \"kubernetes.io/empty-dir/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-bundle\") pod \"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87\" (UID: \"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87\") " Nov 04 01:18:20.503275 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.503061 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-kube-api-access-vpjxv\" (UniqueName: \"kubernetes.io/projected/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-kube-api-access-vpjxv\") pod \"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87\" (UID: \"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87\") " Nov 04 01:18:20.503720 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.503692 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6196c46f-afbf-46fa-a523-69fe1682f2ec-bundle" pod "6196c46f-afbf-46fa-a523-69fe1682f2ec" (UID: "6196c46f-afbf-46fa-a523-69fe1682f2ec"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:18:20.504877 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.504827 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" podStartSLOduration=2.385276894 podStartE2EDuration="25.504811408s" totalImagesPullingTime="23.119534514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:20.393343075 +0000 UTC m=+450.425076476" watchObservedRunningTime="2026-11-04 01:18:20.504811408 +0000 UTC m=+450.536544799" Nov 04 01:18:20.505440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.505322 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11af4777-1f84-4d59-a8f0-5abdd866f88d-bundle" pod "11af4777-1f84-4d59-a8f0-5abdd866f88d" (UID: "11af4777-1f84-4d59-a8f0-5abdd866f88d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:18:20.505727 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.505688 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-bundle" pod "07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87" (UID: "07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:18:20.508756 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.506883 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-kube-api-access-vpjxv" pod "07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87" (UID: "07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87"). InnerVolumeSpecName "kube-api-access-vpjxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:18:20.508756 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.507834 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6196c46f-afbf-46fa-a523-69fe1682f2ec-kube-api-access-9fcb2" pod "6196c46f-afbf-46fa-a523-69fe1682f2ec" (UID: "6196c46f-afbf-46fa-a523-69fe1682f2ec"). InnerVolumeSpecName "kube-api-access-9fcb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:18:20.512044 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.512020 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11af4777-1f84-4d59-a8f0-5abdd866f88d-kube-api-access-d76nf" pod "11af4777-1f84-4d59-a8f0-5abdd866f88d" (UID: "11af4777-1f84-4d59-a8f0-5abdd866f88d"). InnerVolumeSpecName "kube-api-access-d76nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:18:20.512912 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.512886 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-util" pod "07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87" (UID: "07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:18:20.517246 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.517221 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6196c46f-afbf-46fa-a523-69fe1682f2ec-util" pod "6196c46f-afbf-46fa-a523-69fe1682f2ec" (UID: "6196c46f-afbf-46fa-a523-69fe1682f2ec"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:18:20.518511 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.518482 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11af4777-1f84-4d59-a8f0-5abdd866f88d-util" pod "11af4777-1f84-4d59-a8f0-5abdd866f88d" (UID: "11af4777-1f84-4d59-a8f0-5abdd866f88d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:18:20.603984 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.603934 5995 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11af4777-1f84-4d59-a8f0-5abdd866f88d-util\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:20.603984 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.603957 5995 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6196c46f-afbf-46fa-a523-69fe1682f2ec-bundle\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:20.603984 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.603967 5995 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6196c46f-afbf-46fa-a523-69fe1682f2ec-util\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:20.603984 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.603976 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fcb2\" (UniqueName: \"kubernetes.io/projected/6196c46f-afbf-46fa-a523-69fe1682f2ec-kube-api-access-9fcb2\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:20.603984 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.603985 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d76nf\" (UniqueName: \"kubernetes.io/projected/11af4777-1f84-4d59-a8f0-5abdd866f88d-kube-api-access-d76nf\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:20.604184 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.603993 5995 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-util\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:20.604184 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.604001 5995 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11af4777-1f84-4d59-a8f0-5abdd866f88d-bundle\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:20.604184 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.604010 5995 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-bundle\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:20.604184 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:20.604018 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpjxv\" (UniqueName: \"kubernetes.io/projected/07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87-kube-api-access-vpjxv\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:21.198129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:21.198103 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" Nov 04 01:18:21.198542 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:21.198123 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8zzbc" event={"ID":"11af4777-1f84-4d59-a8f0-5abdd866f88d","Type":"ContainerDied","Data":"2bb213444aa7a66727c7894986cce5169094b61e4308eabb13da4d91fbdb8286"} Nov 04 01:18:21.198542 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:21.198168 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bb213444aa7a66727c7894986cce5169094b61e4308eabb13da4d91fbdb8286" Nov 04 01:18:21.200080 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:21.200060 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn" Nov 04 01:18:21.200195 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:21.200054 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9lkfn" event={"ID":"07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87","Type":"ContainerDied","Data":"92da3aa3bf980bd4f3f713f7e760032dc509c4f11462b641add987e82c8eb2b6"} Nov 04 01:18:21.200195 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:21.200184 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92da3aa3bf980bd4f3f713f7e760032dc509c4f11462b641add987e82c8eb2b6" Nov 04 01:18:21.202314 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:21.202290 5995 generic.go:356] "Generic (PLEG): container finished" podID="ec1abf28-0e48-45ab-a58e-c6bd3459e8c7" containerID="e1733d557d1cfa21051f8aba8529a96c95c1513717e495095a4d82ef547fdcb7" exitCode=0 Nov 04 01:18:21.202442 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:21.202387 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn" event={"ID":"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7","Type":"ContainerDied","Data":"e1733d557d1cfa21051f8aba8529a96c95c1513717e495095a4d82ef547fdcb7"} Nov 04 01:18:21.204349 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:21.204333 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" Nov 04 01:18:21.204436 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:21.204346 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71npss2" event={"ID":"6196c46f-afbf-46fa-a523-69fe1682f2ec","Type":"ContainerDied","Data":"2f65d54a18e291d58d973ba1d2b85ddbc6c75d1fab1bf238167b9ed0c2d11469"} Nov 04 01:18:21.204436 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:21.204380 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f65d54a18e291d58d973ba1d2b85ddbc6c75d1fab1bf238167b9ed0c2d11469" Nov 04 01:18:21.206494 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:21.206472 5995 generic.go:356] "Generic (PLEG): container finished" podID="f342ad90-5804-4f04-9ca1-991bdec69858" containerID="71a659f7b40ae78cfbe14a82bb5b3b17051c48ffa3aa4eafdbe8e6e7f3c31472" exitCode=0 Nov 04 01:18:21.206594 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:21.206511 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb" event={"ID":"f342ad90-5804-4f04-9ca1-991bdec69858","Type":"ContainerDied","Data":"71a659f7b40ae78cfbe14a82bb5b3b17051c48ffa3aa4eafdbe8e6e7f3c31472"} Nov 04 01:18:22.348095 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.348062 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb" Nov 04 01:18:22.372156 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.372134 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn" Nov 04 01:18:22.416399 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.416368 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-bundle\") pod \"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7\" (UID: \"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7\") " Nov 04 01:18:22.416533 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.416445 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f342ad90-5804-4f04-9ca1-991bdec69858-util\" (UniqueName: \"kubernetes.io/empty-dir/f342ad90-5804-4f04-9ca1-991bdec69858-util\") pod \"f342ad90-5804-4f04-9ca1-991bdec69858\" (UID: \"f342ad90-5804-4f04-9ca1-991bdec69858\") " Nov 04 01:18:22.416533 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.416496 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f342ad90-5804-4f04-9ca1-991bdec69858-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f342ad90-5804-4f04-9ca1-991bdec69858-bundle\") pod \"f342ad90-5804-4f04-9ca1-991bdec69858\" (UID: \"f342ad90-5804-4f04-9ca1-991bdec69858\") " Nov 04 01:18:22.416656 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.416591 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-kube-api-access-s4brp\" (UniqueName: \"kubernetes.io/projected/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-kube-api-access-s4brp\") pod \"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7\" (UID: \"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7\") " Nov 04 01:18:22.416656 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.416645 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-util\" (UniqueName: \"kubernetes.io/empty-dir/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-util\") pod \"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7\" (UID: \"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7\") " Nov 04 01:18:22.416775 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.416697 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f342ad90-5804-4f04-9ca1-991bdec69858-kube-api-access-4zptj\" (UniqueName: \"kubernetes.io/projected/f342ad90-5804-4f04-9ca1-991bdec69858-kube-api-access-4zptj\") pod \"f342ad90-5804-4f04-9ca1-991bdec69858\" (UID: \"f342ad90-5804-4f04-9ca1-991bdec69858\") " Nov 04 01:18:22.417087 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.417054 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f342ad90-5804-4f04-9ca1-991bdec69858-bundle" pod "f342ad90-5804-4f04-9ca1-991bdec69858" (UID: "f342ad90-5804-4f04-9ca1-991bdec69858"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:18:22.417197 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.417170 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-bundle" pod "ec1abf28-0e48-45ab-a58e-c6bd3459e8c7" (UID: "ec1abf28-0e48-45ab-a58e-c6bd3459e8c7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:18:22.418855 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.418825 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-kube-api-access-s4brp" pod "ec1abf28-0e48-45ab-a58e-c6bd3459e8c7" (UID: "ec1abf28-0e48-45ab-a58e-c6bd3459e8c7"). InnerVolumeSpecName "kube-api-access-s4brp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:18:22.419069 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.419048 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f342ad90-5804-4f04-9ca1-991bdec69858-kube-api-access-4zptj" pod "f342ad90-5804-4f04-9ca1-991bdec69858" (UID: "f342ad90-5804-4f04-9ca1-991bdec69858"). InnerVolumeSpecName "kube-api-access-4zptj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:18:22.422424 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.422395 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-util" pod "ec1abf28-0e48-45ab-a58e-c6bd3459e8c7" (UID: "ec1abf28-0e48-45ab-a58e-c6bd3459e8c7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:18:22.422623 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.422605 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f342ad90-5804-4f04-9ca1-991bdec69858-util" pod "f342ad90-5804-4f04-9ca1-991bdec69858" (UID: "f342ad90-5804-4f04-9ca1-991bdec69858"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:18:22.517397 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.517321 5995 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-bundle\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:22.517397 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.517345 5995 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f342ad90-5804-4f04-9ca1-991bdec69858-util\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:22.517397 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.517354 5995 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f342ad90-5804-4f04-9ca1-991bdec69858-bundle\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:22.517397 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.517363 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4brp\" (UniqueName: \"kubernetes.io/projected/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-kube-api-access-s4brp\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:22.517397 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.517374 5995 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec1abf28-0e48-45ab-a58e-c6bd3459e8c7-util\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:22.517397 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:22.517383 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zptj\" (UniqueName: \"kubernetes.io/projected/f342ad90-5804-4f04-9ca1-991bdec69858-kube-api-access-4zptj\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:23.216725 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:23.216693 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb" Nov 04 01:18:23.216725 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:23.216724 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77vdzb" event={"ID":"f342ad90-5804-4f04-9ca1-991bdec69858","Type":"ContainerDied","Data":"22c72322b56ab0f2d208207f5f84ce3d1fc049a9f503e58bd26f5ffbf1bef91e"} Nov 04 01:18:23.216970 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:23.216777 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22c72322b56ab0f2d208207f5f84ce3d1fc049a9f503e58bd26f5ffbf1bef91e" Nov 04 01:18:23.218528 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:23.218504 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn" event={"ID":"ec1abf28-0e48-45ab-a58e-c6bd3459e8c7","Type":"ContainerDied","Data":"59368006c2385305a69aca13b79cdcdab361ee1ef5fd4b65e36dd5f95c8c4694"} Nov 04 01:18:23.218528 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:23.218523 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cksdrn" Nov 04 01:18:23.218675 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:23.218544 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59368006c2385305a69aca13b79cdcdab361ee1ef5fd4b65e36dd5f95c8c4694" Nov 04 01:18:26.474221 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474185 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65"] Nov 04 01:18:26.474725 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474703 5995 cpu_manager.go:414] "containerMap: removing container" podUID="11af4777-1f84-4d59-a8f0-5abdd866f88d" containerName="extract" Nov 04 01:18:26.474792 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474728 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11af4777-1f84-4d59-a8f0-5abdd866f88d" containerName="extract" Nov 04 01:18:26.474792 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474749 5995 cpu_manager.go:414] "containerMap: removing container" podUID="ec1abf28-0e48-45ab-a58e-c6bd3459e8c7" containerName="extract" Nov 04 01:18:26.474792 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474758 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1abf28-0e48-45ab-a58e-c6bd3459e8c7" containerName="extract" Nov 04 01:18:26.474792 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474770 5995 cpu_manager.go:414] "containerMap: removing container" podUID="6196c46f-afbf-46fa-a523-69fe1682f2ec" containerName="util" Nov 04 01:18:26.474792 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474777 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6196c46f-afbf-46fa-a523-69fe1682f2ec" containerName="util" Nov 04 01:18:26.474792 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474787 5995 cpu_manager.go:414] "containerMap: removing container" podUID="11af4777-1f84-4d59-a8f0-5abdd866f88d" containerName="util" Nov 04 01:18:26.474792 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474793 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11af4777-1f84-4d59-a8f0-5abdd866f88d" containerName="util" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474807 5995 cpu_manager.go:414] "containerMap: removing container" podUID="ec1abf28-0e48-45ab-a58e-c6bd3459e8c7" containerName="util" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474814 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1abf28-0e48-45ab-a58e-c6bd3459e8c7" containerName="util" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474823 5995 cpu_manager.go:414] "containerMap: removing container" podUID="07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87" containerName="util" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474828 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87" containerName="util" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474847 5995 cpu_manager.go:414] "containerMap: removing container" podUID="6196c46f-afbf-46fa-a523-69fe1682f2ec" containerName="pull" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474854 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6196c46f-afbf-46fa-a523-69fe1682f2ec" containerName="pull" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474864 5995 cpu_manager.go:414] "containerMap: removing container" podUID="11af4777-1f84-4d59-a8f0-5abdd866f88d" containerName="pull" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474870 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11af4777-1f84-4d59-a8f0-5abdd866f88d" containerName="pull" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474881 5995 cpu_manager.go:414] "containerMap: removing container" podUID="ec1abf28-0e48-45ab-a58e-c6bd3459e8c7" containerName="pull" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474888 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1abf28-0e48-45ab-a58e-c6bd3459e8c7" containerName="pull" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474898 5995 cpu_manager.go:414] "containerMap: removing container" podUID="6196c46f-afbf-46fa-a523-69fe1682f2ec" containerName="extract" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474904 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6196c46f-afbf-46fa-a523-69fe1682f2ec" containerName="extract" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474914 5995 cpu_manager.go:414] "containerMap: removing container" podUID="07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87" containerName="extract" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474921 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87" containerName="extract" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474933 5995 cpu_manager.go:414] "containerMap: removing container" podUID="f342ad90-5804-4f04-9ca1-991bdec69858" containerName="pull" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474940 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f342ad90-5804-4f04-9ca1-991bdec69858" containerName="pull" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474955 5995 cpu_manager.go:414] "containerMap: removing container" podUID="f342ad90-5804-4f04-9ca1-991bdec69858" containerName="util" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474962 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f342ad90-5804-4f04-9ca1-991bdec69858" containerName="util" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474988 5995 cpu_manager.go:414] "containerMap: removing container" podUID="07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87" containerName="pull" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.474995 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87" containerName="pull" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.475004 5995 cpu_manager.go:414] "containerMap: removing container" podUID="f342ad90-5804-4f04-9ca1-991bdec69858" containerName="extract" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.475012 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f342ad90-5804-4f04-9ca1-991bdec69858" containerName="extract" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.475103 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="6196c46f-afbf-46fa-a523-69fe1682f2ec" containerName="extract" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.475116 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f9b4f6-6bcd-4f0f-9c9a-10425c2dfe87" containerName="extract" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.475129 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="f342ad90-5804-4f04-9ca1-991bdec69858" containerName="extract" Nov 04 01:18:26.475129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.475138 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="11af4777-1f84-4d59-a8f0-5abdd866f88d" containerName="extract" Nov 04 01:18:26.476267 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.475147 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1abf28-0e48-45ab-a58e-c6bd3459e8c7" containerName="extract" Nov 04 01:18:26.507397 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.507367 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.550511 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.550479 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a63a1d5-5148-4377-8281-c4c8e21298cd-tmp\") pod \"opendatahub-operator-controller-manager-5bdd5c57b4-bgs65\" (UID: \"4a63a1d5-5148-4377-8281-c4c8e21298cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.550649 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.550525 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a63a1d5-5148-4377-8281-c4c8e21298cd-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bdd5c57b4-bgs65\" (UID: \"4a63a1d5-5148-4377-8281-c4c8e21298cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.550649 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.550546 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqz28\" (UniqueName: \"kubernetes.io/projected/4a63a1d5-5148-4377-8281-c4c8e21298cd-kube-api-access-xqz28\") pod \"opendatahub-operator-controller-manager-5bdd5c57b4-bgs65\" (UID: \"4a63a1d5-5148-4377-8281-c4c8e21298cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.550720 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.550667 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4a63a1d5-5148-4377-8281-c4c8e21298cd-manifests\") pod \"opendatahub-operator-controller-manager-5bdd5c57b4-bgs65\" (UID: \"4a63a1d5-5148-4377-8281-c4c8e21298cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.550754 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.550721 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a63a1d5-5148-4377-8281-c4c8e21298cd-metrics-cert\") pod \"opendatahub-operator-controller-manager-5bdd5c57b4-bgs65\" (UID: \"4a63a1d5-5148-4377-8281-c4c8e21298cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.550754 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.550738 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4a63a1d5-5148-4377-8281-c4c8e21298cd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bdd5c57b4-bgs65\" (UID: \"4a63a1d5-5148-4377-8281-c4c8e21298cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.592884 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.592857 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 04 01:18:26.592884 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.592871 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 04 01:18:26.593056 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.592876 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 04 01:18:26.603433 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.603395 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 04 01:18:26.603433 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.603399 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 04 01:18:26.603588 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.603435 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-7sbgk\"" Nov 04 01:18:26.608566 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.608546 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65"] Nov 04 01:18:26.652070 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.652039 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a63a1d5-5148-4377-8281-c4c8e21298cd-metrics-cert\") pod \"opendatahub-operator-controller-manager-5bdd5c57b4-bgs65\" (UID: \"4a63a1d5-5148-4377-8281-c4c8e21298cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.652252 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.652075 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4a63a1d5-5148-4377-8281-c4c8e21298cd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bdd5c57b4-bgs65\" (UID: \"4a63a1d5-5148-4377-8281-c4c8e21298cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.652252 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.652118 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a63a1d5-5148-4377-8281-c4c8e21298cd-tmp\") pod \"opendatahub-operator-controller-manager-5bdd5c57b4-bgs65\" (UID: \"4a63a1d5-5148-4377-8281-c4c8e21298cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.652252 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.652160 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a63a1d5-5148-4377-8281-c4c8e21298cd-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bdd5c57b4-bgs65\" (UID: \"4a63a1d5-5148-4377-8281-c4c8e21298cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.652252 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.652191 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqz28\" (UniqueName: \"kubernetes.io/projected/4a63a1d5-5148-4377-8281-c4c8e21298cd-kube-api-access-xqz28\") pod \"opendatahub-operator-controller-manager-5bdd5c57b4-bgs65\" (UID: \"4a63a1d5-5148-4377-8281-c4c8e21298cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.652252 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.652224 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4a63a1d5-5148-4377-8281-c4c8e21298cd-manifests\") pod \"opendatahub-operator-controller-manager-5bdd5c57b4-bgs65\" (UID: \"4a63a1d5-5148-4377-8281-c4c8e21298cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.652703 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.652659 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4a63a1d5-5148-4377-8281-c4c8e21298cd-manifests\") pod \"opendatahub-operator-controller-manager-5bdd5c57b4-bgs65\" (UID: \"4a63a1d5-5148-4377-8281-c4c8e21298cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.654721 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.654697 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a63a1d5-5148-4377-8281-c4c8e21298cd-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bdd5c57b4-bgs65\" (UID: \"4a63a1d5-5148-4377-8281-c4c8e21298cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.654847 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.654730 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a63a1d5-5148-4377-8281-c4c8e21298cd-tmp\") pod \"opendatahub-operator-controller-manager-5bdd5c57b4-bgs65\" (UID: \"4a63a1d5-5148-4377-8281-c4c8e21298cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.654847 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.654800 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4a63a1d5-5148-4377-8281-c4c8e21298cd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bdd5c57b4-bgs65\" (UID: \"4a63a1d5-5148-4377-8281-c4c8e21298cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.655057 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.655034 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a63a1d5-5148-4377-8281-c4c8e21298cd-metrics-cert\") pod \"opendatahub-operator-controller-manager-5bdd5c57b4-bgs65\" (UID: \"4a63a1d5-5148-4377-8281-c4c8e21298cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.713393 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.713363 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqz28\" (UniqueName: \"kubernetes.io/projected/4a63a1d5-5148-4377-8281-c4c8e21298cd-kube-api-access-xqz28\") pod \"opendatahub-operator-controller-manager-5bdd5c57b4-bgs65\" (UID: \"4a63a1d5-5148-4377-8281-c4c8e21298cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:26.855537 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:26.855448 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:27.442625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:27.442591 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65"] Nov 04 01:18:27.450343 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:18:27.450307 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a63a1d5_5148_4377_8281_c4c8e21298cd.slice/crio-9beaabc75d9a75d96a2120fc0a6a3fd55bea0c5a201713489b7f0db1e3f97f00 WatchSource:0}: Error finding container 9beaabc75d9a75d96a2120fc0a6a3fd55bea0c5a201713489b7f0db1e3f97f00: Status 404 returned error can't find the container with id 9beaabc75d9a75d96a2120fc0a6a3fd55bea0c5a201713489b7f0db1e3f97f00 Nov 04 01:18:28.241973 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:28.241896 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" event={"ID":"4a63a1d5-5148-4377-8281-c4c8e21298cd","Type":"ContainerStarted","Data":"9beaabc75d9a75d96a2120fc0a6a3fd55bea0c5a201713489b7f0db1e3f97f00"} Nov 04 01:18:30.253083 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:30.253047 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f_5600ddae-56c7-4c93-afff-2900207b0819/extract/0.log" Nov 04 01:18:30.254088 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:30.254051 5995 generic.go:356] "Generic (PLEG): container finished" podID="5600ddae-56c7-4c93-afff-2900207b0819" containerID="579bed695c48e1e0687f0670a5f7d3f2162507f498924190f716a29be410ad34" exitCode=1 Nov 04 01:18:30.254217 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:30.254115 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" event={"ID":"5600ddae-56c7-4c93-afff-2900207b0819","Type":"ContainerDied","Data":"579bed695c48e1e0687f0670a5f7d3f2162507f498924190f716a29be410ad34"} Nov 04 01:18:31.384235 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:31.384211 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f_5600ddae-56c7-4c93-afff-2900207b0819/extract/0.log" Nov 04 01:18:31.398137 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:31.398111 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" Nov 04 01:18:31.496787 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:31.496756 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5600ddae-56c7-4c93-afff-2900207b0819-kube-api-access-zpc6d\" (UniqueName: \"kubernetes.io/projected/5600ddae-56c7-4c93-afff-2900207b0819-kube-api-access-zpc6d\") pod \"5600ddae-56c7-4c93-afff-2900207b0819\" (UID: \"5600ddae-56c7-4c93-afff-2900207b0819\") " Nov 04 01:18:31.496972 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:31.496814 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5600ddae-56c7-4c93-afff-2900207b0819-util\" (UniqueName: \"kubernetes.io/empty-dir/5600ddae-56c7-4c93-afff-2900207b0819-util\") pod \"5600ddae-56c7-4c93-afff-2900207b0819\" (UID: \"5600ddae-56c7-4c93-afff-2900207b0819\") " Nov 04 01:18:31.496972 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:31.496841 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5600ddae-56c7-4c93-afff-2900207b0819-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5600ddae-56c7-4c93-afff-2900207b0819-bundle\") pod \"5600ddae-56c7-4c93-afff-2900207b0819\" (UID: \"5600ddae-56c7-4c93-afff-2900207b0819\") " Nov 04 01:18:31.497277 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:31.497247 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5600ddae-56c7-4c93-afff-2900207b0819-bundle" pod "5600ddae-56c7-4c93-afff-2900207b0819" (UID: "5600ddae-56c7-4c93-afff-2900207b0819"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:18:31.498895 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:31.498874 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5600ddae-56c7-4c93-afff-2900207b0819-kube-api-access-zpc6d" pod "5600ddae-56c7-4c93-afff-2900207b0819" (UID: "5600ddae-56c7-4c93-afff-2900207b0819"). InnerVolumeSpecName "kube-api-access-zpc6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:18:31.502180 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:31.502155 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5600ddae-56c7-4c93-afff-2900207b0819-util" pod "5600ddae-56c7-4c93-afff-2900207b0819" (UID: "5600ddae-56c7-4c93-afff-2900207b0819"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:18:31.598446 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:31.598356 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpc6d\" (UniqueName: \"kubernetes.io/projected/5600ddae-56c7-4c93-afff-2900207b0819-kube-api-access-zpc6d\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:31.598446 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:31.598387 5995 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5600ddae-56c7-4c93-afff-2900207b0819-util\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:31.598446 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:31.598396 5995 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5600ddae-56c7-4c93-afff-2900207b0819-bundle\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:32.265351 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:32.265323 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f_5600ddae-56c7-4c93-afff-2900207b0819/extract/0.log" Nov 04 01:18:32.266102 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:32.266073 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" Nov 04 01:18:32.266238 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:32.266073 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f" event={"ID":"5600ddae-56c7-4c93-afff-2900207b0819","Type":"ContainerDied","Data":"1b36906c849d9516aa0acad273c827d461927cd1df9c9ed1ca63731291084304"} Nov 04 01:18:32.266238 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:32.266192 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b36906c849d9516aa0acad273c827d461927cd1df9c9ed1ca63731291084304" Nov 04 01:18:32.267809 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:32.267777 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" event={"ID":"4a63a1d5-5148-4377-8281-c4c8e21298cd","Type":"ContainerStarted","Data":"c4d8c551ee06a733d823ab1ae1fbb946e3a49ab9bbb0d7361bd540e69ec2123e"} Nov 04 01:18:33.274019 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:33.273988 5995 generic.go:356] "Generic (PLEG): container finished" podID="4a63a1d5-5148-4377-8281-c4c8e21298cd" containerID="c4d8c551ee06a733d823ab1ae1fbb946e3a49ab9bbb0d7361bd540e69ec2123e" exitCode=0 Nov 04 01:18:33.274449 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:33.274072 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" event={"ID":"4a63a1d5-5148-4377-8281-c4c8e21298cd","Type":"ContainerDied","Data":"c4d8c551ee06a733d823ab1ae1fbb946e3a49ab9bbb0d7361bd540e69ec2123e"} Nov 04 01:18:34.280346 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:34.280308 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" event={"ID":"4a63a1d5-5148-4377-8281-c4c8e21298cd","Type":"ContainerStarted","Data":"4764f8f6bb9d4c5188f3e7904bb45c479491a0b601838a4b884beae85780e650"} Nov 04 01:18:34.280826 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:34.280460 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:34.504099 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:34.504039 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" podStartSLOduration=2.956003294 podStartE2EDuration="8.504025311s" totalImagesPullingTime="5.102810995s" totalInitContainerRuntime="445.211022ms" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:34.457992822 +0000 UTC m=+464.489726215" watchObservedRunningTime="2026-11-04 01:18:34.504025311 +0000 UTC m=+464.535758772" Nov 04 01:18:43.244746 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.244708 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9"] Nov 04 01:18:43.245110 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.245057 5995 cpu_manager.go:414] "containerMap: removing container" podUID="5600ddae-56c7-4c93-afff-2900207b0819" containerName="extract" Nov 04 01:18:43.245110 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.245070 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5600ddae-56c7-4c93-afff-2900207b0819" containerName="extract" Nov 04 01:18:43.245110 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.245082 5995 cpu_manager.go:414] "containerMap: removing container" podUID="5600ddae-56c7-4c93-afff-2900207b0819" containerName="util" Nov 04 01:18:43.245110 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.245087 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5600ddae-56c7-4c93-afff-2900207b0819" containerName="util" Nov 04 01:18:43.245110 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.245104 5995 cpu_manager.go:414] "containerMap: removing container" podUID="5600ddae-56c7-4c93-afff-2900207b0819" containerName="pull" Nov 04 01:18:43.245110 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.245109 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5600ddae-56c7-4c93-afff-2900207b0819" containerName="pull" Nov 04 01:18:43.245307 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.245163 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="5600ddae-56c7-4c93-afff-2900207b0819" containerName="extract" Nov 04 01:18:43.248950 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.248930 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9" Nov 04 01:18:43.298705 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.298676 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9"] Nov 04 01:18:43.343710 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.343678 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 04 01:18:43.343710 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.343688 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:18:43.343912 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.343830 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-pbcwx\"" Nov 04 01:18:43.403999 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.403968 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b887a355-10b3-4256-bb7b-57af9291ceba-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9\" (UID: \"b887a355-10b3-4256-bb7b-57af9291ceba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9" Nov 04 01:18:43.404150 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.404095 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mbzb\" (UniqueName: \"kubernetes.io/projected/b887a355-10b3-4256-bb7b-57af9291ceba-kube-api-access-6mbzb\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9\" (UID: \"b887a355-10b3-4256-bb7b-57af9291ceba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9" Nov 04 01:18:43.505011 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.504920 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mbzb\" (UniqueName: \"kubernetes.io/projected/b887a355-10b3-4256-bb7b-57af9291ceba-kube-api-access-6mbzb\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9\" (UID: \"b887a355-10b3-4256-bb7b-57af9291ceba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9" Nov 04 01:18:43.505011 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.504963 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b887a355-10b3-4256-bb7b-57af9291ceba-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9\" (UID: \"b887a355-10b3-4256-bb7b-57af9291ceba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9" Nov 04 01:18:43.505311 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.505296 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b887a355-10b3-4256-bb7b-57af9291ceba-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9\" (UID: \"b887a355-10b3-4256-bb7b-57af9291ceba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9" Nov 04 01:18:43.565622 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.565586 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mbzb\" (UniqueName: \"kubernetes.io/projected/b887a355-10b3-4256-bb7b-57af9291ceba-kube-api-access-6mbzb\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9\" (UID: \"b887a355-10b3-4256-bb7b-57af9291ceba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9" Nov 04 01:18:43.859309 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:43.859229 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9" Nov 04 01:18:44.329860 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:44.329829 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9"] Nov 04 01:18:44.336682 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:18:44.336645 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb887a355_10b3_4256_bb7b_57af9291ceba.slice/crio-8ad3b93fa5c1ca169a367c6ca206f11538da2219d047de100f8f43bb38676633 WatchSource:0}: Error finding container 8ad3b93fa5c1ca169a367c6ca206f11538da2219d047de100f8f43bb38676633: Status 404 returned error can't find the container with id 8ad3b93fa5c1ca169a367c6ca206f11538da2219d047de100f8f43bb38676633 Nov 04 01:18:45.287550 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:45.287518 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5bdd5c57b4-bgs65" Nov 04 01:18:45.331844 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:45.331812 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9" event={"ID":"b887a355-10b3-4256-bb7b-57af9291ceba","Type":"ContainerStarted","Data":"8ad3b93fa5c1ca169a367c6ca206f11538da2219d047de100f8f43bb38676633"} Nov 04 01:18:46.011123 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:46.011092 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lzpmm"] Nov 04 01:18:46.014130 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:46.014099 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lzpmm" Nov 04 01:18:46.066636 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:46.066598 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 04 01:18:46.066636 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:46.066599 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9r2dn\"" Nov 04 01:18:46.066636 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:46.066627 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 04 01:18:46.075163 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:46.075136 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lzpmm"] Nov 04 01:18:46.126702 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:46.126668 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c282fd58-c58f-4a46-9e63-ab07b0eb2069-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lzpmm\" (UID: \"c282fd58-c58f-4a46-9e63-ab07b0eb2069\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lzpmm" Nov 04 01:18:46.126873 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:46.126797 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cdqc\" (UniqueName: \"kubernetes.io/projected/c282fd58-c58f-4a46-9e63-ab07b0eb2069-kube-api-access-5cdqc\") pod \"kuadrant-operator-controller-manager-678d87766f-lzpmm\" (UID: \"c282fd58-c58f-4a46-9e63-ab07b0eb2069\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lzpmm" Nov 04 01:18:46.227595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:46.227554 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cdqc\" (UniqueName: \"kubernetes.io/projected/c282fd58-c58f-4a46-9e63-ab07b0eb2069-kube-api-access-5cdqc\") pod \"kuadrant-operator-controller-manager-678d87766f-lzpmm\" (UID: \"c282fd58-c58f-4a46-9e63-ab07b0eb2069\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lzpmm" Nov 04 01:18:46.227767 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:46.227603 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c282fd58-c58f-4a46-9e63-ab07b0eb2069-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lzpmm\" (UID: \"c282fd58-c58f-4a46-9e63-ab07b0eb2069\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lzpmm" Nov 04 01:18:46.227966 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:46.227949 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c282fd58-c58f-4a46-9e63-ab07b0eb2069-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lzpmm\" (UID: \"c282fd58-c58f-4a46-9e63-ab07b0eb2069\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lzpmm" Nov 04 01:18:46.286109 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:46.286041 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cdqc\" (UniqueName: \"kubernetes.io/projected/c282fd58-c58f-4a46-9e63-ab07b0eb2069-kube-api-access-5cdqc\") pod \"kuadrant-operator-controller-manager-678d87766f-lzpmm\" (UID: \"c282fd58-c58f-4a46-9e63-ab07b0eb2069\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lzpmm" Nov 04 01:18:46.326461 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:46.326402 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lzpmm" Nov 04 01:18:46.747769 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:46.747730 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lzpmm"] Nov 04 01:18:46.754441 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:18:46.754391 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc282fd58_c58f_4a46_9e63_ab07b0eb2069.slice/crio-77d38ada5e38ff8d6363d9fbc7ab7d1fdaf07b006ebeaa14b3e8a9cf929ada7b WatchSource:0}: Error finding container 77d38ada5e38ff8d6363d9fbc7ab7d1fdaf07b006ebeaa14b3e8a9cf929ada7b: Status 404 returned error can't find the container with id 77d38ada5e38ff8d6363d9fbc7ab7d1fdaf07b006ebeaa14b3e8a9cf929ada7b Nov 04 01:18:47.349605 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:47.349569 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lzpmm" event={"ID":"c282fd58-c58f-4a46-9e63-ab07b0eb2069","Type":"ContainerStarted","Data":"77d38ada5e38ff8d6363d9fbc7ab7d1fdaf07b006ebeaa14b3e8a9cf929ada7b"} Nov 04 01:18:48.356045 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:48.356007 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9" event={"ID":"b887a355-10b3-4256-bb7b-57af9291ceba","Type":"ContainerStarted","Data":"5a5cabe53be6ccf3063e936cac21901a705435e3d49dcf9e9ddeccf019c9fa97"} Nov 04 01:18:48.563335 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:48.563270 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jdvg9" podStartSLOduration=2.409296709 podStartE2EDuration="5.56325159s" totalImagesPullingTime="3.153954881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:48.515185065 +0000 UTC m=+478.546918458" watchObservedRunningTime="2026-11-04 01:18:48.56325159 +0000 UTC m=+478.594984982" Nov 04 01:18:49.652964 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:18:49.652924 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:18:52.353753 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:52.353726 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6qnlj"] Nov 04 01:18:52.356926 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:52.356906 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6qnlj" Nov 04 01:18:52.407375 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:52.407349 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-k98xw\"" Nov 04 01:18:52.425708 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:52.425676 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6qnlj"] Nov 04 01:18:52.426606 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:52.426581 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 04 01:18:52.487400 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:52.487367 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jkf6\" (UniqueName: \"kubernetes.io/projected/5062a83f-5161-447e-a161-72f1422286d2-kube-api-access-8jkf6\") pod \"dns-operator-controller-manager-8557cbdb84-6qnlj\" (UID: \"5062a83f-5161-447e-a161-72f1422286d2\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6qnlj" Nov 04 01:18:52.588162 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:52.588124 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jkf6\" (UniqueName: \"kubernetes.io/projected/5062a83f-5161-447e-a161-72f1422286d2-kube-api-access-8jkf6\") pod \"dns-operator-controller-manager-8557cbdb84-6qnlj\" (UID: \"5062a83f-5161-447e-a161-72f1422286d2\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6qnlj" Nov 04 01:18:52.645775 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:52.645709 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jkf6\" (UniqueName: \"kubernetes.io/projected/5062a83f-5161-447e-a161-72f1422286d2-kube-api-access-8jkf6\") pod \"dns-operator-controller-manager-8557cbdb84-6qnlj\" (UID: \"5062a83f-5161-447e-a161-72f1422286d2\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6qnlj" Nov 04 01:18:52.689265 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:52.689231 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6qnlj" Nov 04 01:18:53.055899 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:53.055862 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6qnlj"] Nov 04 01:18:53.062032 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:18:53.062007 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5062a83f_5161_447e_a161_72f1422286d2.slice/crio-8009678293cee5bd86c0d2ef5878e2fd2f7a1a7e4ccfcb2885d38064700b26a3 WatchSource:0}: Error finding container 8009678293cee5bd86c0d2ef5878e2fd2f7a1a7e4ccfcb2885d38064700b26a3: Status 404 returned error can't find the container with id 8009678293cee5bd86c0d2ef5878e2fd2f7a1a7e4ccfcb2885d38064700b26a3 Nov 04 01:18:53.383634 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:53.383533 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lzpmm" event={"ID":"c282fd58-c58f-4a46-9e63-ab07b0eb2069","Type":"ContainerStarted","Data":"37ec320cf4990a8ae3db7cf5246211b74758bb6f6c582a6f28159b16a7f8c796"} Nov 04 01:18:53.384050 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:53.383701 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lzpmm" Nov 04 01:18:53.384928 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:53.384905 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6qnlj" event={"ID":"5062a83f-5161-447e-a161-72f1422286d2","Type":"ContainerStarted","Data":"8009678293cee5bd86c0d2ef5878e2fd2f7a1a7e4ccfcb2885d38064700b26a3"} Nov 04 01:18:53.496838 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:53.496787 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lzpmm" podStartSLOduration=2.907387946 podStartE2EDuration="8.496774099s" totalImagesPullingTime="5.589386153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:53.495568724 +0000 UTC m=+483.527302116" watchObservedRunningTime="2026-11-04 01:18:53.496774099 +0000 UTC m=+483.528507488" Nov 04 01:18:56.859851 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:56.859766 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pbz49"] Nov 04 01:18:56.862240 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:56.862219 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pbz49" Nov 04 01:18:56.925799 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:56.925770 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4e005554-58f5-4e78-9611-043b8d3ccd6a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pbz49\" (UID: \"4e005554-58f5-4e78-9611-043b8d3ccd6a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pbz49" Nov 04 01:18:56.925953 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:56.925806 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4tzw\" (UniqueName: \"kubernetes.io/projected/4e005554-58f5-4e78-9611-043b8d3ccd6a-kube-api-access-d4tzw\") pod \"kuadrant-console-plugin-66c4fb575d-pbz49\" (UID: \"4e005554-58f5-4e78-9611-043b8d3ccd6a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pbz49" Nov 04 01:18:56.925953 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:56.925830 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e005554-58f5-4e78-9611-043b8d3ccd6a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pbz49\" (UID: \"4e005554-58f5-4e78-9611-043b8d3ccd6a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pbz49" Nov 04 01:18:56.957635 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:56.957607 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4v8rp\"" Nov 04 01:18:56.957802 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:56.957664 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 04 01:18:56.957802 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:56.957709 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 04 01:18:56.959338 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:56.959318 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pbz49"] Nov 04 01:18:57.026705 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:57.026673 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4e005554-58f5-4e78-9611-043b8d3ccd6a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pbz49\" (UID: \"4e005554-58f5-4e78-9611-043b8d3ccd6a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pbz49" Nov 04 01:18:57.026705 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:57.026709 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4tzw\" (UniqueName: \"kubernetes.io/projected/4e005554-58f5-4e78-9611-043b8d3ccd6a-kube-api-access-d4tzw\") pod \"kuadrant-console-plugin-66c4fb575d-pbz49\" (UID: \"4e005554-58f5-4e78-9611-043b8d3ccd6a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pbz49" Nov 04 01:18:57.026954 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:57.026730 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e005554-58f5-4e78-9611-043b8d3ccd6a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pbz49\" (UID: \"4e005554-58f5-4e78-9611-043b8d3ccd6a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pbz49" Nov 04 01:18:57.027437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:57.027400 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4e005554-58f5-4e78-9611-043b8d3ccd6a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pbz49\" (UID: \"4e005554-58f5-4e78-9611-043b8d3ccd6a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pbz49" Nov 04 01:18:57.029076 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:57.029058 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e005554-58f5-4e78-9611-043b8d3ccd6a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pbz49\" (UID: \"4e005554-58f5-4e78-9611-043b8d3ccd6a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pbz49" Nov 04 01:18:57.084884 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:57.084845 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4tzw\" (UniqueName: \"kubernetes.io/projected/4e005554-58f5-4e78-9611-043b8d3ccd6a-kube-api-access-d4tzw\") pod \"kuadrant-console-plugin-66c4fb575d-pbz49\" (UID: \"4e005554-58f5-4e78-9611-043b8d3ccd6a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pbz49" Nov 04 01:18:57.189602 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:57.189505 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pbz49" Nov 04 01:18:57.405605 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:57.405573 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6qnlj" event={"ID":"5062a83f-5161-447e-a161-72f1422286d2","Type":"ContainerStarted","Data":"88951e65c5cfc4dfadc048f1f3d01de38d5de9a2d69fcd926ed53dce8f57e65a"} Nov 04 01:18:57.405775 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:57.405656 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6qnlj" Nov 04 01:18:57.520634 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:57.520581 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6qnlj" podStartSLOduration=2.023535373 podStartE2EDuration="5.520565613s" totalImagesPullingTime="3.49703024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:57.518900598 +0000 UTC m=+487.550633990" watchObservedRunningTime="2026-11-04 01:18:57.520565613 +0000 UTC m=+487.552298997" Nov 04 01:18:57.555448 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:57.555404 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pbz49"] Nov 04 01:18:57.562173 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:18:57.562140 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e005554_58f5_4e78_9611_043b8d3ccd6a.slice/crio-315f3f4f2562bc2c8cdcb2ac26bfcf81cf60b936d91794e113f63cadeb29e926 WatchSource:0}: Error finding container 315f3f4f2562bc2c8cdcb2ac26bfcf81cf60b936d91794e113f63cadeb29e926: Status 404 returned error can't find the container with id 315f3f4f2562bc2c8cdcb2ac26bfcf81cf60b936d91794e113f63cadeb29e926 Nov 04 01:18:58.411245 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:18:58.411209 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pbz49" event={"ID":"4e005554-58f5-4e78-9611-043b8d3ccd6a","Type":"ContainerStarted","Data":"315f3f4f2562bc2c8cdcb2ac26bfcf81cf60b936d91794e113f63cadeb29e926"} Nov 04 01:19:04.392095 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:04.392061 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lzpmm" Nov 04 01:19:04.444342 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:04.444303 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pbz49" event={"ID":"4e005554-58f5-4e78-9611-043b8d3ccd6a","Type":"ContainerStarted","Data":"544a215111504f4df2269e9f2f2781172fc4bbdb64e3af81558ab4853511a0d1"} Nov 04 01:19:04.659683 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:04.659587 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pbz49" podStartSLOduration=2.448909216 podStartE2EDuration="8.659570842s" totalImagesPullingTime="6.210661626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:04.658402751 +0000 UTC m=+494.690136141" watchObservedRunningTime="2026-11-04 01:19:04.659570842 +0000 UTC m=+494.691304228" Nov 04 01:19:08.414087 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:08.414051 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6qnlj" Nov 04 01:19:19.639935 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:19:19.639894 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:19:26.371362 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:26.371328 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54cfc47d9b-5tzlw"] Nov 04 01:19:26.419766 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:26.419742 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54cfc47d9b-5tzlw" Nov 04 01:19:26.431171 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:26.431143 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54cfc47d9b-5tzlw"] Nov 04 01:19:26.485345 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:26.485315 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c2206b8-9e89-423c-92f7-a7ed330d7421-tmp\") pod \"kube-auth-proxy-54cfc47d9b-5tzlw\" (UID: \"2c2206b8-9e89-423c-92f7-a7ed330d7421\") " pod="openshift-ingress/kube-auth-proxy-54cfc47d9b-5tzlw" Nov 04 01:19:26.485510 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:26.485356 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c2206b8-9e89-423c-92f7-a7ed330d7421-tls-certs\") pod \"kube-auth-proxy-54cfc47d9b-5tzlw\" (UID: \"2c2206b8-9e89-423c-92f7-a7ed330d7421\") " pod="openshift-ingress/kube-auth-proxy-54cfc47d9b-5tzlw" Nov 04 01:19:26.485510 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:26.485477 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-825sl\" (UniqueName: \"kubernetes.io/projected/2c2206b8-9e89-423c-92f7-a7ed330d7421-kube-api-access-825sl\") pod \"kube-auth-proxy-54cfc47d9b-5tzlw\" (UID: \"2c2206b8-9e89-423c-92f7-a7ed330d7421\") " pod="openshift-ingress/kube-auth-proxy-54cfc47d9b-5tzlw" Nov 04 01:19:26.523687 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:26.523652 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 04 01:19:26.523852 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:26.523837 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 04 01:19:26.523905 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:26.523852 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-qszhl\"" Nov 04 01:19:26.586550 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:26.586520 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c2206b8-9e89-423c-92f7-a7ed330d7421-tmp\") pod \"kube-auth-proxy-54cfc47d9b-5tzlw\" (UID: \"2c2206b8-9e89-423c-92f7-a7ed330d7421\") " pod="openshift-ingress/kube-auth-proxy-54cfc47d9b-5tzlw" Nov 04 01:19:26.586550 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:26.586554 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c2206b8-9e89-423c-92f7-a7ed330d7421-tls-certs\") pod \"kube-auth-proxy-54cfc47d9b-5tzlw\" (UID: \"2c2206b8-9e89-423c-92f7-a7ed330d7421\") " pod="openshift-ingress/kube-auth-proxy-54cfc47d9b-5tzlw" Nov 04 01:19:26.586773 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:26.586584 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-825sl\" (UniqueName: \"kubernetes.io/projected/2c2206b8-9e89-423c-92f7-a7ed330d7421-kube-api-access-825sl\") pod \"kube-auth-proxy-54cfc47d9b-5tzlw\" (UID: \"2c2206b8-9e89-423c-92f7-a7ed330d7421\") " pod="openshift-ingress/kube-auth-proxy-54cfc47d9b-5tzlw" Nov 04 01:19:26.588880 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:26.588855 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c2206b8-9e89-423c-92f7-a7ed330d7421-tmp\") pod \"kube-auth-proxy-54cfc47d9b-5tzlw\" (UID: \"2c2206b8-9e89-423c-92f7-a7ed330d7421\") " pod="openshift-ingress/kube-auth-proxy-54cfc47d9b-5tzlw" Nov 04 01:19:26.589969 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:26.589945 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c2206b8-9e89-423c-92f7-a7ed330d7421-tls-certs\") pod \"kube-auth-proxy-54cfc47d9b-5tzlw\" (UID: \"2c2206b8-9e89-423c-92f7-a7ed330d7421\") " pod="openshift-ingress/kube-auth-proxy-54cfc47d9b-5tzlw" Nov 04 01:19:26.648920 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:26.648850 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-825sl\" (UniqueName: \"kubernetes.io/projected/2c2206b8-9e89-423c-92f7-a7ed330d7421-kube-api-access-825sl\") pod \"kube-auth-proxy-54cfc47d9b-5tzlw\" (UID: \"2c2206b8-9e89-423c-92f7-a7ed330d7421\") " pod="openshift-ingress/kube-auth-proxy-54cfc47d9b-5tzlw" Nov 04 01:19:26.747690 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:26.747657 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54cfc47d9b-5tzlw" Nov 04 01:19:27.132244 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:27.132211 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54cfc47d9b-5tzlw"] Nov 04 01:19:27.138738 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:19:27.138712 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2206b8_9e89_423c_92f7_a7ed330d7421.slice/crio-30af7b32af80ba01ed2d499e5f2c177be8feede72f679ef6c5690b890d282ba3 WatchSource:0}: Error finding container 30af7b32af80ba01ed2d499e5f2c177be8feede72f679ef6c5690b890d282ba3: Status 404 returned error can't find the container with id 30af7b32af80ba01ed2d499e5f2c177be8feede72f679ef6c5690b890d282ba3 Nov 04 01:19:27.552940 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:27.552906 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54cfc47d9b-5tzlw" event={"ID":"2c2206b8-9e89-423c-92f7-a7ed330d7421","Type":"ContainerStarted","Data":"30af7b32af80ba01ed2d499e5f2c177be8feede72f679ef6c5690b890d282ba3"} Nov 04 01:19:30.570270 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:30.570237 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54cfc47d9b-5tzlw" event={"ID":"2c2206b8-9e89-423c-92f7-a7ed330d7421","Type":"ContainerStarted","Data":"e03b3f505a8fc214426ce7b62c16dfdd42f409c84130ceefecdb460ef45a8063"} Nov 04 01:19:30.679068 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:30.679014 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54cfc47d9b-5tzlw" podStartSLOduration=1.678602214 podStartE2EDuration="4.679000368s" totalImagesPullingTime="3.000398154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:30.678835133 +0000 UTC m=+520.710568519" watchObservedRunningTime="2026-11-04 01:19:30.679000368 +0000 UTC m=+520.710733758" Nov 04 01:19:31.987685 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:31.987647 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:19:31.990832 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:31.990809 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-jf7b5" Nov 04 01:19:32.039863 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:32.039831 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 04 01:19:32.046446 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:32.046406 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:19:32.137691 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:32.137653 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww925\" (UniqueName: \"kubernetes.io/projected/d1825e94-4a7d-4ea5-ad61-b1b4fa486373-kube-api-access-ww925\") pod \"limitador-limitador-7c9b87d574-jf7b5\" (UID: \"d1825e94-4a7d-4ea5-ad61-b1b4fa486373\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jf7b5" Nov 04 01:19:32.137867 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:32.137756 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d1825e94-4a7d-4ea5-ad61-b1b4fa486373-config-file\") pod \"limitador-limitador-7c9b87d574-jf7b5\" (UID: \"d1825e94-4a7d-4ea5-ad61-b1b4fa486373\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jf7b5" Nov 04 01:19:32.238567 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:32.238493 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d1825e94-4a7d-4ea5-ad61-b1b4fa486373-config-file\") pod \"limitador-limitador-7c9b87d574-jf7b5\" (UID: \"d1825e94-4a7d-4ea5-ad61-b1b4fa486373\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jf7b5" Nov 04 01:19:32.238752 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:32.238653 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww925\" (UniqueName: \"kubernetes.io/projected/d1825e94-4a7d-4ea5-ad61-b1b4fa486373-kube-api-access-ww925\") pod \"limitador-limitador-7c9b87d574-jf7b5\" (UID: \"d1825e94-4a7d-4ea5-ad61-b1b4fa486373\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jf7b5" Nov 04 01:19:32.244482 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:32.240729 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d1825e94-4a7d-4ea5-ad61-b1b4fa486373-config-file\") pod \"limitador-limitador-7c9b87d574-jf7b5\" (UID: \"d1825e94-4a7d-4ea5-ad61-b1b4fa486373\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jf7b5" Nov 04 01:19:32.272777 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:32.272748 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:19:32.297869 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:32.297837 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww925\" (UniqueName: \"kubernetes.io/projected/d1825e94-4a7d-4ea5-ad61-b1b4fa486373-kube-api-access-ww925\") pod \"limitador-limitador-7c9b87d574-jf7b5\" (UID: \"d1825e94-4a7d-4ea5-ad61-b1b4fa486373\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jf7b5" Nov 04 01:19:32.301797 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:32.301772 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-jf7b5" Nov 04 01:19:32.668597 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:32.668517 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:19:32.675523 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:19:32.675491 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1825e94_4a7d_4ea5_ad61_b1b4fa486373.slice/crio-b40af9812183d979d8ee28ab9d82da48a94076404c3fb816052918f7ae03650a WatchSource:0}: Error finding container b40af9812183d979d8ee28ab9d82da48a94076404c3fb816052918f7ae03650a: Status 404 returned error can't find the container with id b40af9812183d979d8ee28ab9d82da48a94076404c3fb816052918f7ae03650a Nov 04 01:19:33.585630 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:33.585591 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-jf7b5" event={"ID":"d1825e94-4a7d-4ea5-ad61-b1b4fa486373","Type":"ContainerStarted","Data":"b40af9812183d979d8ee28ab9d82da48a94076404c3fb816052918f7ae03650a"} Nov 04 01:19:35.596669 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:35.596581 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-jf7b5" event={"ID":"d1825e94-4a7d-4ea5-ad61-b1b4fa486373","Type":"ContainerStarted","Data":"40f3a0667452dd4987d7ecbf9e20134c380ed849564a2f3d39f4095f16a59037"} Nov 04 01:19:35.596669 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:35.596649 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-jf7b5" Nov 04 01:19:35.705801 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:35.705652 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-jf7b5" podStartSLOduration=2.186222506 podStartE2EDuration="4.705635168s" totalImagesPullingTime="2.519412662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:35.705378079 +0000 UTC m=+525.737111469" watchObservedRunningTime="2026-11-04 01:19:35.705635168 +0000 UTC m=+525.737368559" Nov 04 01:19:46.603047 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:19:46.603016 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-jf7b5" Nov 04 01:19:49.640917 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:19:49.640879 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:20:19.641576 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:20:19.641495 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:20:49.640577 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:20:49.640536 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:20:50.160507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:50.160481 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f_5600ddae-56c7-4c93-afff-2900207b0819/extract/0.log" Nov 04 01:20:50.160722 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:50.160534 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f_5600ddae-56c7-4c93-afff-2900207b0819/extract/0.log" Nov 04 01:20:50.169752 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:50.169730 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:20:50.169889 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:50.169862 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:20:58.814082 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:58.814048 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm"] Nov 04 01:20:58.817500 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:58.817481 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:58.866626 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:58.866595 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 04 01:20:58.866626 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:58.866595 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 04 01:20:58.866626 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:58.866608 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 04 01:20:58.866839 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:58.866830 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-b44fv\"" Nov 04 01:20:58.916291 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:58.916264 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/51aec6f1-b80d-498b-b762-892e032f4999-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:58.916471 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:58.916305 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51aec6f1-b80d-498b-b762-892e032f4999-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:58.916471 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:58.916332 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/51aec6f1-b80d-498b-b762-892e032f4999-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:58.916593 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:58.916520 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51aec6f1-b80d-498b-b762-892e032f4999-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:58.916593 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:58.916543 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/51aec6f1-b80d-498b-b762-892e032f4999-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:58.916593 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:58.916574 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/51aec6f1-b80d-498b-b762-892e032f4999-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:58.916735 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:58.916594 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zz6p\" (UniqueName: \"kubernetes.io/projected/51aec6f1-b80d-498b-b762-892e032f4999-kube-api-access-7zz6p\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:58.917276 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:58.917246 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm"] Nov 04 01:20:59.017577 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:59.017549 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51aec6f1-b80d-498b-b762-892e032f4999-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:59.017773 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:59.017593 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/51aec6f1-b80d-498b-b762-892e032f4999-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:59.017773 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:59.017619 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/51aec6f1-b80d-498b-b762-892e032f4999-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:59.017773 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:59.017640 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zz6p\" (UniqueName: \"kubernetes.io/projected/51aec6f1-b80d-498b-b762-892e032f4999-kube-api-access-7zz6p\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:59.017773 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:59.017717 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/51aec6f1-b80d-498b-b762-892e032f4999-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:59.017773 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:59.017749 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51aec6f1-b80d-498b-b762-892e032f4999-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:59.017773 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:59.017771 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/51aec6f1-b80d-498b-b762-892e032f4999-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:59.018761 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:59.018728 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/51aec6f1-b80d-498b-b762-892e032f4999-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:59.020180 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:59.020153 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/51aec6f1-b80d-498b-b762-892e032f4999-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:59.020273 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:59.020190 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51aec6f1-b80d-498b-b762-892e032f4999-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:59.020475 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:59.020447 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/51aec6f1-b80d-498b-b762-892e032f4999-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:59.020674 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:59.020655 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/51aec6f1-b80d-498b-b762-892e032f4999-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:59.075064 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:59.075003 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51aec6f1-b80d-498b-b762-892e032f4999-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:59.075176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:59.075100 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zz6p\" (UniqueName: \"kubernetes.io/projected/51aec6f1-b80d-498b-b762-892e032f4999-kube-api-access-7zz6p\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gfwmm\" (UID: \"51aec6f1-b80d-498b-b762-892e032f4999\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:59.128936 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:59.128911 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:20:59.549794 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:20:59.549758 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm"] Nov 04 01:20:59.555882 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:20:59.555856 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51aec6f1_b80d_498b_b762_892e032f4999.slice/crio-f2d5deac95865ef5c06e8c7bc859db948f149bc9b7db843163829c2a62ea6a49 WatchSource:0}: Error finding container f2d5deac95865ef5c06e8c7bc859db948f149bc9b7db843163829c2a62ea6a49: Status 404 returned error can't find the container with id f2d5deac95865ef5c06e8c7bc859db948f149bc9b7db843163829c2a62ea6a49 Nov 04 01:21:00.024890 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:21:00.024859 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" event={"ID":"51aec6f1-b80d-498b-b762-892e032f4999","Type":"ContainerStarted","Data":"f2d5deac95865ef5c06e8c7bc859db948f149bc9b7db843163829c2a62ea6a49"} Nov 04 01:21:03.711984 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:21:03.711952 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:21:04.522201 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:21:04.522170 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:21:04.522313 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:21:04.522234 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:21:05.053345 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:21:05.053312 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" event={"ID":"51aec6f1-b80d-498b-b762-892e032f4999","Type":"ContainerStarted","Data":"ea2b98d4495fd6c217581fcaaed2fc67ec2c5c8d3fb23f84942ef87192b75f8d"} Nov 04 01:21:05.053810 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:21:05.053553 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:21:05.055067 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:21:05.055043 5995 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-gfwmm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:21:05.055191 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:21:05.055093 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" podUID="51aec6f1-b80d-498b-b762-892e032f4999" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:21:05.169184 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:21:05.169127 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" podStartSLOduration=2.205664218 podStartE2EDuration="7.169111783s" totalImagesPullingTime="4.963447565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:05.1679557 +0000 UTC m=+615.199689090" watchObservedRunningTime="2026-11-04 01:21:05.169111783 +0000 UTC m=+615.200845173" Nov 04 01:21:06.058246 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:21:06.058219 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gfwmm" Nov 04 01:21:19.641396 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:21:19.641358 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:21:49.640994 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:21:49.640956 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:22:19.640449 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:22:19.640399 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:22:49.640493 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:22:49.640456 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:23:19.641142 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:23:19.641061 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:23:49.641911 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:23:49.641874 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:23:58.934010 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:23:58.933974 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:24:06.585529 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:06.585493 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:24:19.640538 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:24:19.640501 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:24:21.401962 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.401927 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:24:21.600343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.600309 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8"] Nov 04 01:24:21.603285 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.603262 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.652638 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.652570 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 04 01:24:21.652638 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.652570 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 04 01:24:21.697300 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.697256 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4141f1e2-a242-463e-977b-87ca0ba3b369-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8\" (UID: \"4141f1e2-a242-463e-977b-87ca0ba3b369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.697444 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.697331 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4141f1e2-a242-463e-977b-87ca0ba3b369-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8\" (UID: \"4141f1e2-a242-463e-977b-87ca0ba3b369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.697444 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.697355 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k46g4\" (UniqueName: \"kubernetes.io/projected/4141f1e2-a242-463e-977b-87ca0ba3b369-kube-api-access-k46g4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8\" (UID: \"4141f1e2-a242-463e-977b-87ca0ba3b369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.697444 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.697427 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4141f1e2-a242-463e-977b-87ca0ba3b369-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8\" (UID: \"4141f1e2-a242-463e-977b-87ca0ba3b369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.697554 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.697454 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4141f1e2-a242-463e-977b-87ca0ba3b369-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8\" (UID: \"4141f1e2-a242-463e-977b-87ca0ba3b369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.697554 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.697512 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4141f1e2-a242-463e-977b-87ca0ba3b369-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8\" (UID: \"4141f1e2-a242-463e-977b-87ca0ba3b369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.699078 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.699063 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kccc6\"" Nov 04 01:24:21.699332 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.699315 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 04 01:24:21.700943 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.700925 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8"] Nov 04 01:24:21.798866 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.798828 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4141f1e2-a242-463e-977b-87ca0ba3b369-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8\" (UID: \"4141f1e2-a242-463e-977b-87ca0ba3b369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.798866 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.798868 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k46g4\" (UniqueName: \"kubernetes.io/projected/4141f1e2-a242-463e-977b-87ca0ba3b369-kube-api-access-k46g4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8\" (UID: \"4141f1e2-a242-463e-977b-87ca0ba3b369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.799117 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.798989 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4141f1e2-a242-463e-977b-87ca0ba3b369-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8\" (UID: \"4141f1e2-a242-463e-977b-87ca0ba3b369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.799117 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.799034 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4141f1e2-a242-463e-977b-87ca0ba3b369-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8\" (UID: \"4141f1e2-a242-463e-977b-87ca0ba3b369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.799117 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.799073 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4141f1e2-a242-463e-977b-87ca0ba3b369-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8\" (UID: \"4141f1e2-a242-463e-977b-87ca0ba3b369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.799297 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.799165 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4141f1e2-a242-463e-977b-87ca0ba3b369-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8\" (UID: \"4141f1e2-a242-463e-977b-87ca0ba3b369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.799379 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.799354 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4141f1e2-a242-463e-977b-87ca0ba3b369-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8\" (UID: \"4141f1e2-a242-463e-977b-87ca0ba3b369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.799466 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.799375 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4141f1e2-a242-463e-977b-87ca0ba3b369-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8\" (UID: \"4141f1e2-a242-463e-977b-87ca0ba3b369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.799602 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.799583 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4141f1e2-a242-463e-977b-87ca0ba3b369-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8\" (UID: \"4141f1e2-a242-463e-977b-87ca0ba3b369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.801743 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.801713 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4141f1e2-a242-463e-977b-87ca0ba3b369-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8\" (UID: \"4141f1e2-a242-463e-977b-87ca0ba3b369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.801910 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.801891 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4141f1e2-a242-463e-977b-87ca0ba3b369-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8\" (UID: \"4141f1e2-a242-463e-977b-87ca0ba3b369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.857272 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.857242 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k46g4\" (UniqueName: \"kubernetes.io/projected/4141f1e2-a242-463e-977b-87ca0ba3b369-kube-api-access-k46g4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8\" (UID: \"4141f1e2-a242-463e-977b-87ca0ba3b369\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:21.936958 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:21.936877 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:22.313885 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:22.313854 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8"] Nov 04 01:24:22.320452 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:24:22.320408 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4141f1e2_a242_463e_977b_87ca0ba3b369.slice/crio-ff3c492c643cdfeaf3fbed42bc1c587a005e5ffa71a17afdfe25ab5349f2ecf8 WatchSource:0}: Error finding container ff3c492c643cdfeaf3fbed42bc1c587a005e5ffa71a17afdfe25ab5349f2ecf8: Status 404 returned error can't find the container with id ff3c492c643cdfeaf3fbed42bc1c587a005e5ffa71a17afdfe25ab5349f2ecf8 Nov 04 01:24:22.321714 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:22.321699 5995 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:24:23.021778 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.021734 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" event={"ID":"4141f1e2-a242-463e-977b-87ca0ba3b369","Type":"ContainerStarted","Data":"ff3c492c643cdfeaf3fbed42bc1c587a005e5ffa71a17afdfe25ab5349f2ecf8"} Nov 04 01:24:23.520140 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.520107 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24"] Nov 04 01:24:23.522944 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.522921 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.572238 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.572201 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kcfvk\"" Nov 04 01:24:23.616646 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.616614 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.616646 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.616646 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4de5e6-8906-4510-ab46-f5842d546a8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.616844 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.616669 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.616844 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.616777 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.616913 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.616848 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.616913 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.616890 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.616973 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.616915 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.617005 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.616986 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nmcp\" (UniqueName: \"kubernetes.io/projected/ec4de5e6-8906-4510-ab46-f5842d546a8a-kube-api-access-6nmcp\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.617036 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.617010 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.622868 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.622829 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24"] Nov 04 01:24:23.717569 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.717534 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.717718 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.717591 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.717718 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.717621 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.717718 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.717652 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nmcp\" (UniqueName: \"kubernetes.io/projected/ec4de5e6-8906-4510-ab46-f5842d546a8a-kube-api-access-6nmcp\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.717718 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.717684 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.717718 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.717713 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.717948 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.717737 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4de5e6-8906-4510-ab46-f5842d546a8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.717948 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.717771 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.717948 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.717812 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.718105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.718005 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.718105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.718095 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.718218 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.718114 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.718280 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.718234 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.718548 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.718526 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4de5e6-8906-4510-ab46-f5842d546a8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.719961 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.719939 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.720206 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.720189 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.777049 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.776974 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.777189 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.777099 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nmcp\" (UniqueName: \"kubernetes.io/projected/ec4de5e6-8906-4510-ab46-f5842d546a8a-kube-api-access-6nmcp\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qch24\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:23.835331 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:23.835297 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:24.259265 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:24.259229 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24"] Nov 04 01:24:24.265952 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:24:24.265926 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4de5e6_8906_4510_ab46_f5842d546a8a.slice/crio-c9268b06f1426f2b4564c00a14214a92a504f84d8ce811e33e185263ff821f7a WatchSource:0}: Error finding container c9268b06f1426f2b4564c00a14214a92a504f84d8ce811e33e185263ff821f7a: Status 404 returned error can't find the container with id c9268b06f1426f2b4564c00a14214a92a504f84d8ce811e33e185263ff821f7a Nov 04 01:24:24.268035 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:24.268001 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:24:24.268124 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:24.268067 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:24:24.268124 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:24.268093 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:24:25.034867 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:25.034824 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" event={"ID":"ec4de5e6-8906-4510-ab46-f5842d546a8a","Type":"ContainerStarted","Data":"927d887a998ee346e70a4dd4dd2050c32537e8db7ab07bdb3d8a9780257eb151"} Nov 04 01:24:25.035181 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:25.034882 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" event={"ID":"ec4de5e6-8906-4510-ab46-f5842d546a8a","Type":"ContainerStarted","Data":"c9268b06f1426f2b4564c00a14214a92a504f84d8ce811e33e185263ff821f7a"} Nov 04 01:24:25.243433 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:25.243376 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" podStartSLOduration=2.243361504 podStartE2EDuration="2.243361504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:25.195599061 +0000 UTC m=+815.227332484" watchObservedRunningTime="2026-11-04 01:24:25.243361504 +0000 UTC m=+815.275094886" Nov 04 01:24:25.835658 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:25.835618 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:25.837168 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:25.837141 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-qch24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:24:25.837302 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:25.837189 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" podUID="ec4de5e6-8906-4510-ab46-f5842d546a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 04 01:24:26.043082 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:26.043040 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" event={"ID":"4141f1e2-a242-463e-977b-87ca0ba3b369","Type":"ContainerStarted","Data":"752bbf19d58a564b35bede7aa23cf549caf5b8eee1f62286ce584af0261fc329"} Nov 04 01:24:26.156096 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:26.155999 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" podStartSLOduration=2.179003877 podStartE2EDuration="5.155985221s" totalImagesPullingTime="2.976981344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:26.154816798 +0000 UTC m=+816.186550189" watchObservedRunningTime="2026-11-04 01:24:26.155985221 +0000 UTC m=+816.187718611" Nov 04 01:24:26.836170 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:26.836132 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-qch24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:24:26.836598 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:26.836206 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" podUID="ec4de5e6-8906-4510-ab46-f5842d546a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 04 01:24:26.937762 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:26.937730 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:27.836319 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:27.836283 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-qch24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:24:27.836807 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:27.836344 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" podUID="ec4de5e6-8906-4510-ab46-f5842d546a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 04 01:24:28.836458 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:28.836396 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-qch24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:24:28.836959 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:28.836477 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" podUID="ec4de5e6-8906-4510-ab46-f5842d546a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 04 01:24:29.835847 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:29.835808 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-qch24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:24:29.836026 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:29.835881 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" podUID="ec4de5e6-8906-4510-ab46-f5842d546a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 04 01:24:30.836429 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:30.836383 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-qch24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:24:30.836836 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:30.836469 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" podUID="ec4de5e6-8906-4510-ab46-f5842d546a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 04 01:24:31.937784 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:31.937746 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:31.951307 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:31.951278 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:32.038954 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:32.038901 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:32.039369 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:32.039338 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:32.040474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:32.040451 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:24:32.089141 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:32.089116 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8" Nov 04 01:24:37.982831 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:37.982795 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:24:49.641173 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:24:49.641128 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:24:53.776933 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.776895 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8"] Nov 04 01:24:53.779690 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.779660 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.810556 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.810532 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.810691 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.810563 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66684988-7433-4da1-94fb-8110cf7c31c7-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.810691 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.810591 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.810691 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.810662 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h76w\" (UniqueName: \"kubernetes.io/projected/66684988-7433-4da1-94fb-8110cf7c31c7-kube-api-access-5h76w\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.810853 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.810717 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.810853 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.810734 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66684988-7433-4da1-94fb-8110cf7c31c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.810853 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.810836 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.810974 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.810897 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.811030 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.811006 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66684988-7433-4da1-94fb-8110cf7c31c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.870814 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.870781 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8"] Nov 04 01:24:53.911767 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.911728 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.911955 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.911777 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66684988-7433-4da1-94fb-8110cf7c31c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.911955 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.911804 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.911955 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.911822 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66684988-7433-4da1-94fb-8110cf7c31c7-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.911955 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.911847 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.911955 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.911869 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h76w\" (UniqueName: \"kubernetes.io/projected/66684988-7433-4da1-94fb-8110cf7c31c7-kube-api-access-5h76w\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.911955 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.911926 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.911955 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.911956 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66684988-7433-4da1-94fb-8110cf7c31c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.912352 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.911995 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.912442 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.912385 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.912553 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.912470 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.912553 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.912499 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.912637 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.912548 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.912637 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.912614 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66684988-7433-4da1-94fb-8110cf7c31c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.914098 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.914082 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.914431 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.914400 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66684988-7433-4da1-94fb-8110cf7c31c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.972284 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.972248 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66684988-7433-4da1-94fb-8110cf7c31c7-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:53.972284 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:53.972277 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h76w\" (UniqueName: \"kubernetes.io/projected/66684988-7433-4da1-94fb-8110cf7c31c7-kube-api-access-5h76w\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7w5l8\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:54.091882 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:54.091804 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:54.520776 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:54.520742 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8"] Nov 04 01:24:54.527110 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:24:54.527082 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66684988_7433_4da1_94fb_8110cf7c31c7.slice/crio-36ff9035c24e9b2413f3b8eacc65e46ac9e1f1b933349f104fbb7b16920b0150 WatchSource:0}: Error finding container 36ff9035c24e9b2413f3b8eacc65e46ac9e1f1b933349f104fbb7b16920b0150: Status 404 returned error can't find the container with id 36ff9035c24e9b2413f3b8eacc65e46ac9e1f1b933349f104fbb7b16920b0150 Nov 04 01:24:54.529196 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:54.529157 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:24:54.529278 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:54.529226 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:24:54.529278 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:54.529253 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:24:55.193360 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:55.193322 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" event={"ID":"66684988-7433-4da1-94fb-8110cf7c31c7","Type":"ContainerStarted","Data":"268a54db80c8df84011c330a8101e71470887631d5bc801a20ef1d25b03d9c53"} Nov 04 01:24:55.193844 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:55.193376 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" event={"ID":"66684988-7433-4da1-94fb-8110cf7c31c7","Type":"ContainerStarted","Data":"36ff9035c24e9b2413f3b8eacc65e46ac9e1f1b933349f104fbb7b16920b0150"} Nov 04 01:24:55.355099 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:55.355041 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" podStartSLOduration=2.35502461 podStartE2EDuration="2.35502461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:55.353995635 +0000 UTC m=+845.385729041" watchObservedRunningTime="2026-11-04 01:24:55.35502461 +0000 UTC m=+845.386758000" Nov 04 01:24:56.092209 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:56.092172 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:24:56.093787 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:56.093757 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:24:56.093915 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:56.093821 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" podUID="66684988-7433-4da1-94fb-8110cf7c31c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 04 01:24:57.092911 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:57.092873 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:24:57.093311 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:57.092935 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" podUID="66684988-7433-4da1-94fb-8110cf7c31c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 04 01:24:58.092702 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:58.092663 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:24:58.092885 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:58.092721 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" podUID="66684988-7433-4da1-94fb-8110cf7c31c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 04 01:24:59.092729 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:59.092697 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:24:59.093121 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:24:59.092751 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" podUID="66684988-7433-4da1-94fb-8110cf7c31c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 04 01:25:00.093069 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:25:00.093022 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:25:00.093480 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:25:00.093078 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" podUID="66684988-7433-4da1-94fb-8110cf7c31c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 04 01:25:01.092715 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:25:01.092685 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:25:01.092880 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:25:01.092742 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" podUID="66684988-7433-4da1-94fb-8110cf7c31c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 04 01:25:02.823245 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:25:02.823210 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:25:02.823786 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:25:02.823764 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:25:02.824623 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:25:02.824600 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:25:19.641202 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:25:19.641164 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:25:49.641442 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:25:49.641386 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:25:50.198509 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:25:50.198481 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f_5600ddae-56c7-4c93-afff-2900207b0819/extract/0.log" Nov 04 01:25:50.199466 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:25:50.199439 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f_5600ddae-56c7-4c93-afff-2900207b0819/extract/0.log" Nov 04 01:25:50.207796 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:25:50.207777 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:25:50.208493 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:25:50.208474 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:26:10.577266 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.577188 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm"] Nov 04 01:26:10.581013 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.580994 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.631903 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.631868 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9f7183-openshift-default-dockercfg-4jw5f\"" Nov 04 01:26:10.680358 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.680321 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm"] Nov 04 01:26:10.736971 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.736916 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrqbt\" (UniqueName: \"kubernetes.io/projected/7ca12a66-6407-4747-967c-7e31aadc46d2-kube-api-access-qrqbt\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.737159 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.736990 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-credential-socket\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.737159 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.737144 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-workload-certs\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.737343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.737284 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-token\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.737459 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.737430 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-podinfo\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.737534 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.737459 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-data\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.737534 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.737490 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca12a66-6407-4747-967c-7e31aadc46d2-istiod-ca-cert\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.737534 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.737517 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-workload-socket\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.737640 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.737556 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-envoy\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.838577 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.838486 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca12a66-6407-4747-967c-7e31aadc46d2-istiod-ca-cert\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.838577 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.838542 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-workload-socket\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.838783 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.838584 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-envoy\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.838783 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.838606 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrqbt\" (UniqueName: \"kubernetes.io/projected/7ca12a66-6407-4747-967c-7e31aadc46d2-kube-api-access-qrqbt\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.838783 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.838636 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-credential-socket\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.838783 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.838673 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-workload-certs\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.838783 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.838717 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-token\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.838783 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.838766 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-podinfo\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.839025 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.838793 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-data\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.839813 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.839272 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-credential-socket\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.839813 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.839314 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca12a66-6407-4747-967c-7e31aadc46d2-istiod-ca-cert\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.839813 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.839340 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-workload-socket\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.839813 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.839582 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-workload-certs\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.840040 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.839829 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-data\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.841634 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.841614 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-envoy\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.841717 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.841661 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-podinfo\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.896989 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.896955 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-token\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.897153 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.897128 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrqbt\" (UniqueName: \"kubernetes.io/projected/7ca12a66-6407-4747-967c-7e31aadc46d2-kube-api-access-qrqbt\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:10.900985 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:10.900965 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:11.380937 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:11.380899 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm"] Nov 04 01:26:11.386912 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:26:11.386881 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca12a66_6407_4747_967c_7e31aadc46d2.slice/crio-d9664ddf4ed250fb5cec03c61b36be10e7bd58e74dbb20fc063d60abff79e58c WatchSource:0}: Error finding container d9664ddf4ed250fb5cec03c61b36be10e7bd58e74dbb20fc063d60abff79e58c: Status 404 returned error can't find the container with id d9664ddf4ed250fb5cec03c61b36be10e7bd58e74dbb20fc063d60abff79e58c Nov 04 01:26:11.389055 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:11.389025 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:26:11.389133 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:11.389091 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:26:11.389133 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:11.389124 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:26:11.569636 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:11.569598 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" event={"ID":"7ca12a66-6407-4747-967c-7e31aadc46d2","Type":"ContainerStarted","Data":"c485cea884b487821f37a24d2d2ea5907e50e49887b73bad6591a2312ea3a194"} Nov 04 01:26:11.569825 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:11.569659 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" event={"ID":"7ca12a66-6407-4747-967c-7e31aadc46d2","Type":"ContainerStarted","Data":"d9664ddf4ed250fb5cec03c61b36be10e7bd58e74dbb20fc063d60abff79e58c"} Nov 04 01:26:11.684261 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:11.684124 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" podStartSLOduration=1.684102588 podStartE2EDuration="1.684102588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:11.683409356 +0000 UTC m=+921.715142744" watchObservedRunningTime="2026-11-04 01:26:11.684102588 +0000 UTC m=+921.715835978" Nov 04 01:26:11.901327 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:11.901298 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:12.906171 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:12.906139 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:13.580104 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:13.580074 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:13.581024 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:13.581005 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:19.640780 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:26:19.640747 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:26:19.944351 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:19.944309 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2"] Nov 04 01:26:19.949352 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:19.949329 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.003134 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.003097 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b2eadb-openshift-default-dockercfg-b9qd7\"" Nov 04 01:26:20.032171 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.032123 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-credential-socket\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.032339 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.032251 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-envoy\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.032339 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.032314 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-token\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.032441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.032350 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-workload-certs\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.032441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.032386 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istiod-ca-cert\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.032523 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.032482 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-workload-socket\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.032634 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.032616 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-data\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.032679 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.032648 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8jlz\" (UniqueName: \"kubernetes.io/projected/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-kube-api-access-g8jlz\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.032731 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.032676 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-podinfo\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.051468 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.051442 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2"] Nov 04 01:26:20.135827 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.135792 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istiod-ca-cert\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.136000 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.135835 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-workload-socket\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.136000 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.135926 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-data\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.136000 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.135942 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8jlz\" (UniqueName: \"kubernetes.io/projected/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-kube-api-access-g8jlz\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.136000 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.135958 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-podinfo\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.136000 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.135982 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-credential-socket\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.136262 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.136074 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-envoy\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.136262 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.136107 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-token\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.136262 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.136145 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-workload-certs\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.136453 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.136315 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-data\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.136453 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.136368 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-workload-socket\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.136573 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.136524 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-workload-certs\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.136633 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.136609 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istiod-ca-cert\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.136861 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.136840 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-credential-socket\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.138350 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.138330 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-envoy\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.138505 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.138489 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-podinfo\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.193663 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.193628 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-token\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.193663 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.193659 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8jlz\" (UniqueName: \"kubernetes.io/projected/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-kube-api-access-g8jlz\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.263307 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.263231 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:20.688800 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.688720 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2"] Nov 04 01:26:20.730973 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:26:20.730946 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d541a44_f6eb_47ae_b9c5_244f33af1bf1.slice/crio-b9c684c9623011f827fc5eb6d0112c6f4d5f622b10db0280f0a626c4a56539fb WatchSource:0}: Error finding container b9c684c9623011f827fc5eb6d0112c6f4d5f622b10db0280f0a626c4a56539fb: Status 404 returned error can't find the container with id b9c684c9623011f827fc5eb6d0112c6f4d5f622b10db0280f0a626c4a56539fb Nov 04 01:26:20.733177 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.733135 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:26:20.733264 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.733215 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:26:20.733264 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:20.733242 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:26:21.622647 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:21.622614 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" event={"ID":"9d541a44-f6eb-47ae-b9c5-244f33af1bf1","Type":"ContainerStarted","Data":"f019f7d3e19ef088515cc7f289fa2e80ac38c06595c0d04207e2df5be79b32c9"} Nov 04 01:26:21.622821 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:21.622678 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" event={"ID":"9d541a44-f6eb-47ae-b9c5-244f33af1bf1","Type":"ContainerStarted","Data":"b9c684c9623011f827fc5eb6d0112c6f4d5f622b10db0280f0a626c4a56539fb"} Nov 04 01:26:21.737697 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:21.737643 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" podStartSLOduration=2.737628972 podStartE2EDuration="2.737628972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:21.735524696 +0000 UTC m=+931.767258085" watchObservedRunningTime="2026-11-04 01:26:21.737628972 +0000 UTC m=+931.769362362" Nov 04 01:26:22.212084 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.212050 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj"] Nov 04 01:26:22.215991 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.215969 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.260660 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.260624 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-data\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.260824 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.260762 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-envoy\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.260824 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.260799 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-credential-socket\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.260946 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.260837 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-workload-certs\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.260946 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.260910 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-token\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.260946 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.260940 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdpt4\" (UniqueName: \"kubernetes.io/projected/12797bc0-f8ab-4176-a807-1a3918fb8d0e-kube-api-access-rdpt4\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.261099 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.260958 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-workload-socket\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.261099 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.261062 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-podinfo\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.261206 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.261110 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istiod-ca-cert\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.264364 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.264290 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:22.269270 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.269251 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:22.270476 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.270460 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a96a99-openshift-default-dockercfg-hbmfv\"" Nov 04 01:26:22.315729 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.315701 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj"] Nov 04 01:26:22.372437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.362767 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-workload-certs\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.372437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.362959 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-token\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.372437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.363981 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-workload-certs\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.372437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.364045 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdpt4\" (UniqueName: \"kubernetes.io/projected/12797bc0-f8ab-4176-a807-1a3918fb8d0e-kube-api-access-rdpt4\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.372437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.364084 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-workload-socket\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.372437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.364143 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-podinfo\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.372437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.364173 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istiod-ca-cert\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.372437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.364295 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-data\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.372437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.364457 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-envoy\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.372437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.364642 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-data\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.372437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.364990 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istiod-ca-cert\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.372437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.365067 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-workload-socket\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.372437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.365161 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-credential-socket\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.372437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.365303 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-credential-socket\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.372437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.367732 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-podinfo\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.391499 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.390272 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-envoy\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.421009 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.420973 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-token\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.421197 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.421174 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdpt4\" (UniqueName: \"kubernetes.io/projected/12797bc0-f8ab-4176-a807-1a3918fb8d0e-kube-api-access-rdpt4\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.529720 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.529637 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:22.629482 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.629448 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:22.630631 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.630602 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:26:22.953580 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.953548 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj"] Nov 04 01:26:22.959057 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:26:22.959031 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12797bc0_f8ab_4176_a807_1a3918fb8d0e.slice/crio-4c917ade57127bf6cffa24ee7416a48d850ecbdc7d3d8629b9bf4b77dc9e58f9 WatchSource:0}: Error finding container 4c917ade57127bf6cffa24ee7416a48d850ecbdc7d3d8629b9bf4b77dc9e58f9: Status 404 returned error can't find the container with id 4c917ade57127bf6cffa24ee7416a48d850ecbdc7d3d8629b9bf4b77dc9e58f9 Nov 04 01:26:22.961207 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.961174 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:26:22.961296 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.961249 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:26:22.961296 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:22.961277 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:26:23.636078 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:23.636039 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" event={"ID":"12797bc0-f8ab-4176-a807-1a3918fb8d0e","Type":"ContainerStarted","Data":"a5093bcbffa94a7ea26cae640fe5d8922d879ccc9c720d6cc0892aa2afa63591"} Nov 04 01:26:23.636255 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:23.636102 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" event={"ID":"12797bc0-f8ab-4176-a807-1a3918fb8d0e","Type":"ContainerStarted","Data":"4c917ade57127bf6cffa24ee7416a48d850ecbdc7d3d8629b9bf4b77dc9e58f9"} Nov 04 01:26:23.749370 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:23.749321 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" podStartSLOduration=1.74930739 podStartE2EDuration="1.74930739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:23.749289975 +0000 UTC m=+933.781023365" watchObservedRunningTime="2026-11-04 01:26:23.74930739 +0000 UTC m=+933.781040780" Nov 04 01:26:24.530754 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:24.530724 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:24.536075 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:24.536049 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:24.643025 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:24.642992 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:24.643699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:24.643682 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:26:31.752375 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:31.752339 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:26:41.764178 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:41.764143 5995 generic.go:356] "Generic (PLEG): container finished" podID="7ca12a66-6407-4747-967c-7e31aadc46d2" containerID="c485cea884b487821f37a24d2d2ea5907e50e49887b73bad6591a2312ea3a194" exitCode=0 Nov 04 01:26:41.764178 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:41.764182 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" event={"ID":"7ca12a66-6407-4747-967c-7e31aadc46d2","Type":"ContainerDied","Data":"c485cea884b487821f37a24d2d2ea5907e50e49887b73bad6591a2312ea3a194"} Nov 04 01:26:41.764643 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:41.764573 5995 scope.go:122] "RemoveContainer" containerID="c485cea884b487821f37a24d2d2ea5907e50e49887b73bad6591a2312ea3a194" Nov 04 01:26:41.765775 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:41.765746 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:26:41.765839 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:41.765814 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:26:41.765873 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:41.765847 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:26:42.794980 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:42.794947 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" event={"ID":"7ca12a66-6407-4747-967c-7e31aadc46d2","Type":"ContainerStarted","Data":"7f49465316a4b373ff681c8186364c3fce16acf457db45d27e516120640df50a"} Nov 04 01:26:42.901728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:42.901688 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:42.902741 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:42.902709 5995 patch_prober.go:29] interesting pod/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 04 01:26:42.902863 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:42.902771 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" podUID="7ca12a66-6407-4747-967c-7e31aadc46d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 04 01:26:43.586545 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:43.586509 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:43.902436 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:43.902333 5995 patch_prober.go:29] interesting pod/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 04 01:26:43.902436 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:43.902386 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" podUID="7ca12a66-6407-4747-967c-7e31aadc46d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 04 01:26:44.902025 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:44.901993 5995 patch_prober.go:29] interesting pod/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 04 01:26:44.902246 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:44.902041 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" podUID="7ca12a66-6407-4747-967c-7e31aadc46d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 04 01:26:45.907930 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:45.907890 5995 patch_prober.go:29] interesting pod/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 04 01:26:45.909513 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:45.909474 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" podUID="7ca12a66-6407-4747-967c-7e31aadc46d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 04 01:26:46.077017 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:46.076981 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:26:46.901846 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:46.901817 5995 patch_prober.go:29] interesting pod/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 04 01:26:46.902034 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:46.901872 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" podUID="7ca12a66-6407-4747-967c-7e31aadc46d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 04 01:26:46.904567 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:46.904542 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx"] Nov 04 01:26:46.908373 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:46.908356 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" Nov 04 01:26:46.953054 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:46.953024 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxqfx\" (UniqueName: \"kubernetes.io/projected/290fa234-8b22-4932-a683-a782790c388d-kube-api-access-xxqfx\") pod \"payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx\" (UID: \"290fa234-8b22-4932-a683-a782790c388d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" Nov 04 01:26:46.953215 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:46.953167 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290fa234-8b22-4932-a683-a782790c388d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx\" (UID: \"290fa234-8b22-4932-a683-a782790c388d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" Nov 04 01:26:47.011271 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.011232 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-1baf54-dockercfg-fp8wx\"" Nov 04 01:26:47.011860 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.011841 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-1baf54\"" Nov 04 01:26:47.013655 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.013634 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx"] Nov 04 01:26:47.055466 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.054365 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxqfx\" (UniqueName: \"kubernetes.io/projected/290fa234-8b22-4932-a683-a782790c388d-kube-api-access-xxqfx\") pod \"payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx\" (UID: \"290fa234-8b22-4932-a683-a782790c388d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" Nov 04 01:26:47.055466 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.055263 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290fa234-8b22-4932-a683-a782790c388d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx\" (UID: \"290fa234-8b22-4932-a683-a782790c388d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" Nov 04 01:26:47.056238 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.056210 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290fa234-8b22-4932-a683-a782790c388d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx\" (UID: \"290fa234-8b22-4932-a683-a782790c388d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" Nov 04 01:26:47.058208 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.058184 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq"] Nov 04 01:26:47.083461 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.072696 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" Nov 04 01:26:47.112312 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.112284 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxqfx\" (UniqueName: \"kubernetes.io/projected/290fa234-8b22-4932-a683-a782790c388d-kube-api-access-xxqfx\") pod \"payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx\" (UID: \"290fa234-8b22-4932-a683-a782790c388d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" Nov 04 01:26:47.116726 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.116699 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq"] Nov 04 01:26:47.156931 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.156845 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb79d2-6f5f-489f-b80d-de78155993b9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq\" (UID: \"2ecb79d2-6f5f-489f-b80d-de78155993b9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" Nov 04 01:26:47.157100 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.157078 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vshq\" (UniqueName: \"kubernetes.io/projected/2ecb79d2-6f5f-489f-b80d-de78155993b9-kube-api-access-4vshq\") pod \"payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq\" (UID: \"2ecb79d2-6f5f-489f-b80d-de78155993b9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" Nov 04 01:26:47.221116 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.221084 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" Nov 04 01:26:47.258597 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.258565 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vshq\" (UniqueName: \"kubernetes.io/projected/2ecb79d2-6f5f-489f-b80d-de78155993b9-kube-api-access-4vshq\") pod \"payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq\" (UID: \"2ecb79d2-6f5f-489f-b80d-de78155993b9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" Nov 04 01:26:47.258718 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.258649 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb79d2-6f5f-489f-b80d-de78155993b9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq\" (UID: \"2ecb79d2-6f5f-489f-b80d-de78155993b9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" Nov 04 01:26:47.259295 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.259275 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb79d2-6f5f-489f-b80d-de78155993b9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq\" (UID: \"2ecb79d2-6f5f-489f-b80d-de78155993b9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" Nov 04 01:26:47.317481 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.317449 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vshq\" (UniqueName: \"kubernetes.io/projected/2ecb79d2-6f5f-489f-b80d-de78155993b9-kube-api-access-4vshq\") pod \"payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq\" (UID: \"2ecb79d2-6f5f-489f-b80d-de78155993b9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" Nov 04 01:26:47.399779 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.399747 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" Nov 04 01:26:47.620311 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.620277 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx"] Nov 04 01:26:47.627438 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:26:47.627390 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290fa234_8b22_4932_a683_a782790c388d.slice/crio-c010e5df4502e621b9c48fcf3ce08330ecda054f600ec3625b77e131e872501b WatchSource:0}: Error finding container c010e5df4502e621b9c48fcf3ce08330ecda054f600ec3625b77e131e872501b: Status 404 returned error can't find the container with id c010e5df4502e621b9c48fcf3ce08330ecda054f600ec3625b77e131e872501b Nov 04 01:26:47.775510 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.775478 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq"] Nov 04 01:26:47.780629 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:26:47.780596 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecb79d2_6f5f_489f_b80d_de78155993b9.slice/crio-ebbde33b24f9aee1cd3c0e9b8a4692cdd60fc064b8395e7c9a14539564066b38 WatchSource:0}: Error finding container ebbde33b24f9aee1cd3c0e9b8a4692cdd60fc064b8395e7c9a14539564066b38: Status 404 returned error can't find the container with id ebbde33b24f9aee1cd3c0e9b8a4692cdd60fc064b8395e7c9a14539564066b38 Nov 04 01:26:47.822816 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.822782 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" event={"ID":"290fa234-8b22-4932-a683-a782790c388d","Type":"ContainerStarted","Data":"c010e5df4502e621b9c48fcf3ce08330ecda054f600ec3625b77e131e872501b"} Nov 04 01:26:47.823978 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:47.823954 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" event={"ID":"2ecb79d2-6f5f-489f-b80d-de78155993b9","Type":"ContainerStarted","Data":"ebbde33b24f9aee1cd3c0e9b8a4692cdd60fc064b8395e7c9a14539564066b38"} Nov 04 01:26:48.270965 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:48.270934 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:48.272905 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:48.272878 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:26:49.649888 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:26:49.649846 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:26:50.847235 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:50.847196 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" event={"ID":"290fa234-8b22-4932-a683-a782790c388d","Type":"ContainerStarted","Data":"8f118cf833e12f434de0d4e1314c1e07a803dd3c6e6dce59eeafed00ce9581fd"} Nov 04 01:26:50.847617 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:50.847264 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" Nov 04 01:26:50.957167 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:50.957126 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" podStartSLOduration=1.967765232 podStartE2EDuration="4.957109765s" totalImagesPullingTime="2.989344533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:50.956100703 +0000 UTC m=+960.987834093" watchObservedRunningTime="2026-11-04 01:26:50.957109765 +0000 UTC m=+960.988843155" Nov 04 01:26:51.853469 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:51.853424 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" event={"ID":"2ecb79d2-6f5f-489f-b80d-de78155993b9","Type":"ContainerStarted","Data":"505324c77f260825daf96fe9f803b0c07a690177d6de53e49e2edd31c737280e"} Nov 04 01:26:51.853964 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:51.853518 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" Nov 04 01:26:51.964461 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:51.964396 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" podStartSLOduration=1.793513923 podStartE2EDuration="4.964380329s" totalImagesPullingTime="3.170866406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:51.962014791 +0000 UTC m=+961.993748181" watchObservedRunningTime="2026-11-04 01:26:51.964380329 +0000 UTC m=+961.996113720" Nov 04 01:26:55.499885 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:55.499848 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:26:56.711008 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:56.710972 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld"] Nov 04 01:26:56.716000 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:56.715978 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld" Nov 04 01:26:56.791040 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:56.791006 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld"] Nov 04 01:26:56.795844 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:56.795815 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed35833-200d-4911-921f-910721ced9b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld\" (UID: \"0ed35833-200d-4911-921f-910721ced9b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld" Nov 04 01:26:56.795992 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:56.795864 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvvxk\" (UniqueName: \"kubernetes.io/projected/0ed35833-200d-4911-921f-910721ced9b2-kube-api-access-mvvxk\") pod \"payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld\" (UID: \"0ed35833-200d-4911-921f-910721ced9b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld" Nov 04 01:26:56.834310 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:56.834276 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-b2eadb\"" Nov 04 01:26:56.834468 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:56.834352 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-b2eadb-dockercfg-slwq9\"" Nov 04 01:26:56.896659 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:56.896628 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed35833-200d-4911-921f-910721ced9b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld\" (UID: \"0ed35833-200d-4911-921f-910721ced9b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld" Nov 04 01:26:56.896841 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:56.896673 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvvxk\" (UniqueName: \"kubernetes.io/projected/0ed35833-200d-4911-921f-910721ced9b2-kube-api-access-mvvxk\") pod \"payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld\" (UID: \"0ed35833-200d-4911-921f-910721ced9b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld" Nov 04 01:26:56.897531 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:56.897507 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed35833-200d-4911-921f-910721ced9b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld\" (UID: \"0ed35833-200d-4911-921f-910721ced9b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld" Nov 04 01:26:56.954761 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:56.954727 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvvxk\" (UniqueName: \"kubernetes.io/projected/0ed35833-200d-4911-921f-910721ced9b2-kube-api-access-mvvxk\") pod \"payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld\" (UID: \"0ed35833-200d-4911-921f-910721ced9b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld" Nov 04 01:26:57.029064 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:57.028970 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld" Nov 04 01:26:57.420485 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:57.420383 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld"] Nov 04 01:26:57.426624 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:26:57.426595 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed35833_200d_4911_921f_910721ced9b2.slice/crio-6a8e2c25370f4b54eda3ef2336b7ac484900426466a6502db93b0df868501ada WatchSource:0}: Error finding container 6a8e2c25370f4b54eda3ef2336b7ac484900426466a6502db93b0df868501ada: Status 404 returned error can't find the container with id 6a8e2c25370f4b54eda3ef2336b7ac484900426466a6502db93b0df868501ada Nov 04 01:26:57.898933 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:57.898895 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld" event={"ID":"0ed35833-200d-4911-921f-910721ced9b2","Type":"ContainerStarted","Data":"988e0b9d4773b7166382723d677494cff296a6f768cfef69df62ff3e9808f898"} Nov 04 01:26:57.899362 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:57.898944 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld" event={"ID":"0ed35833-200d-4911-921f-910721ced9b2","Type":"ContainerStarted","Data":"6a8e2c25370f4b54eda3ef2336b7ac484900426466a6502db93b0df868501ada"} Nov 04 01:26:57.899362 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:57.899011 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld" Nov 04 01:26:58.008034 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:26:58.007973 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld" podStartSLOduration=2.007955013 podStartE2EDuration="2.007955013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:58.007384945 +0000 UTC m=+968.039118335" watchObservedRunningTime="2026-11-04 01:26:58.007955013 +0000 UTC m=+968.039688403" Nov 04 01:27:01.857214 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:01.857182 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" Nov 04 01:27:02.861990 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:02.861960 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" Nov 04 01:27:05.942523 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:05.942486 5995 generic.go:356] "Generic (PLEG): container finished" podID="12797bc0-f8ab-4176-a807-1a3918fb8d0e" containerID="a5093bcbffa94a7ea26cae640fe5d8922d879ccc9c720d6cc0892aa2afa63591" exitCode=0 Nov 04 01:27:05.943088 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:05.942558 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" event={"ID":"12797bc0-f8ab-4176-a807-1a3918fb8d0e","Type":"ContainerDied","Data":"a5093bcbffa94a7ea26cae640fe5d8922d879ccc9c720d6cc0892aa2afa63591"} Nov 04 01:27:05.943088 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:05.942942 5995 scope.go:122] "RemoveContainer" containerID="a5093bcbffa94a7ea26cae640fe5d8922d879ccc9c720d6cc0892aa2afa63591" Nov 04 01:27:05.944169 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:05.944140 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:27:05.944256 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:05.944207 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:27:05.944256 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:05.944236 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:27:06.950930 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:06.950898 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" event={"ID":"12797bc0-f8ab-4176-a807-1a3918fb8d0e","Type":"ContainerStarted","Data":"8704bc717555d5e2febf5c97e83fe0a870f880f4fb077f90a923f89532391593"} Nov 04 01:27:07.530022 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:07.529989 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:27:07.531207 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:07.531181 5995 patch_prober.go:29] interesting pod/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 04 01:27:07.531342 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:07.531233 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" podUID="12797bc0-f8ab-4176-a807-1a3918fb8d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 04 01:27:08.530047 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:08.530014 5995 patch_prober.go:29] interesting pod/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 04 01:27:08.530467 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:08.530067 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" podUID="12797bc0-f8ab-4176-a807-1a3918fb8d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 04 01:27:08.907702 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:08.907616 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld" Nov 04 01:27:09.530865 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:09.530831 5995 patch_prober.go:29] interesting pod/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 04 01:27:09.531285 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:09.530880 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" podUID="12797bc0-f8ab-4176-a807-1a3918fb8d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 04 01:27:09.646445 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:09.643914 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:27:10.530661 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:10.530621 5995 patch_prober.go:29] interesting pod/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 04 01:27:10.530845 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:10.530695 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" podUID="12797bc0-f8ab-4176-a807-1a3918fb8d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 04 01:27:11.530630 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:11.530597 5995 patch_prober.go:29] interesting pod/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 04 01:27:11.531087 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:11.530657 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" podUID="12797bc0-f8ab-4176-a807-1a3918fb8d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 04 01:27:12.534562 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:12.534531 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:27:12.535580 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:12.535545 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:27:13.693707 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:13.693673 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:27:19.642206 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:27:19.642163 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:27:22.170519 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:22.170483 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:27:32.131784 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:32.131698 5995 generic.go:356] "Generic (PLEG): container finished" podID="9d541a44-f6eb-47ae-b9c5-244f33af1bf1" containerID="f019f7d3e19ef088515cc7f289fa2e80ac38c06595c0d04207e2df5be79b32c9" exitCode=0 Nov 04 01:27:32.132184 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:32.131773 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" event={"ID":"9d541a44-f6eb-47ae-b9c5-244f33af1bf1","Type":"ContainerDied","Data":"f019f7d3e19ef088515cc7f289fa2e80ac38c06595c0d04207e2df5be79b32c9"} Nov 04 01:27:32.132184 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:32.132108 5995 scope.go:122] "RemoveContainer" containerID="f019f7d3e19ef088515cc7f289fa2e80ac38c06595c0d04207e2df5be79b32c9" Nov 04 01:27:32.133310 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:32.133277 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:27:32.133436 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:32.133363 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:27:32.133436 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:32.133407 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:27:33.140210 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:33.140174 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" event={"ID":"9d541a44-f6eb-47ae-b9c5-244f33af1bf1","Type":"ContainerStarted","Data":"04a29f375fa515e212de6fa3ec827a8bcb2cc610bd51e08363bbfc976a2882cf"} Nov 04 01:27:33.273002 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:33.271793 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:27:33.279437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:33.275525 5995 patch_prober.go:29] interesting pod/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 04 01:27:33.279437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:33.275593 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" podUID="9d541a44-f6eb-47ae-b9c5-244f33af1bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 04 01:27:34.284470 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:34.266780 5995 patch_prober.go:29] interesting pod/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 04 01:27:34.284470 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:34.266841 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" podUID="9d541a44-f6eb-47ae-b9c5-244f33af1bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 04 01:27:35.277439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:35.266799 5995 patch_prober.go:29] interesting pod/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 04 01:27:35.277439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:35.266874 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" podUID="9d541a44-f6eb-47ae-b9c5-244f33af1bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 04 01:27:36.274440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:36.269828 5995 patch_prober.go:29] interesting pod/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 04 01:27:36.274440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:36.269896 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" podUID="9d541a44-f6eb-47ae-b9c5-244f33af1bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 04 01:27:37.280439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:37.265759 5995 patch_prober.go:29] interesting pod/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 04 01:27:37.280439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:37.265822 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" podUID="9d541a44-f6eb-47ae-b9c5-244f33af1bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 04 01:27:37.629888 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:37.629794 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:27:38.268869 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:38.268839 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:27:38.269990 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:38.269967 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:27:43.055335 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:43.055297 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:27:49.641606 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:27:49.641571 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:27:52.731582 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.731540 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v"] Nov 04 01:27:52.735807 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.735783 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:52.837160 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.837115 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-00cfe6\"/\"kube-root-ca.crt\"" Nov 04 01:27:52.837160 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.837134 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-00cfe6\"/\"openshift-service-ca.crt\"" Nov 04 01:27:52.839203 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.839182 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v"] Nov 04 01:27:52.885945 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.885733 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-00cfe6\"/\"default-dockercfg-x7gmb\"" Nov 04 01:27:52.885945 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.885900 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-00cfe6\"/\"premium-w2-00cfe6-kserve-self-signed-certs\"" Nov 04 01:27:52.893290 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.892408 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-model-cache\") pod \"premium-w2-00cfe6-kserve-6888bb4c54-b4q4v\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:52.893290 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.892485 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-tmp-dir\") pod \"premium-w2-00cfe6-kserve-6888bb4c54-b4q4v\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:52.893290 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.892568 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-tls-certs\") pod \"premium-w2-00cfe6-kserve-6888bb4c54-b4q4v\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:52.893290 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.892948 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-dshm\") pod \"premium-w2-00cfe6-kserve-6888bb4c54-b4q4v\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:52.893290 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.893036 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hsw5\" (UniqueName: \"kubernetes.io/projected/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-kube-api-access-8hsw5\") pod \"premium-w2-00cfe6-kserve-6888bb4c54-b4q4v\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:52.893290 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.893081 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-home\") pod \"premium-w2-00cfe6-kserve-6888bb4c54-b4q4v\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:52.994598 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.994524 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-tls-certs\") pod \"premium-w2-00cfe6-kserve-6888bb4c54-b4q4v\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:52.994598 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.994578 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-dshm\") pod \"premium-w2-00cfe6-kserve-6888bb4c54-b4q4v\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:52.994821 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.994618 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hsw5\" (UniqueName: \"kubernetes.io/projected/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-kube-api-access-8hsw5\") pod \"premium-w2-00cfe6-kserve-6888bb4c54-b4q4v\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:52.994821 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.994659 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-home\") pod \"premium-w2-00cfe6-kserve-6888bb4c54-b4q4v\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:52.994821 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.994694 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-model-cache\") pod \"premium-w2-00cfe6-kserve-6888bb4c54-b4q4v\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:52.994821 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.994714 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-tmp-dir\") pod \"premium-w2-00cfe6-kserve-6888bb4c54-b4q4v\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:52.995122 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.995093 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-home\") pod \"premium-w2-00cfe6-kserve-6888bb4c54-b4q4v\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:52.995200 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.995159 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-tmp-dir\") pod \"premium-w2-00cfe6-kserve-6888bb4c54-b4q4v\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:52.995261 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.995166 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-model-cache\") pod \"premium-w2-00cfe6-kserve-6888bb4c54-b4q4v\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:52.996927 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.996899 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-dshm\") pod \"premium-w2-00cfe6-kserve-6888bb4c54-b4q4v\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:52.997091 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:52.997076 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-tls-certs\") pod \"premium-w2-00cfe6-kserve-6888bb4c54-b4q4v\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:53.052180 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:53.052143 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hsw5\" (UniqueName: \"kubernetes.io/projected/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-kube-api-access-8hsw5\") pod \"premium-w2-00cfe6-kserve-6888bb4c54-b4q4v\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:53.347581 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:53.347492 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:53.720119 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:53.720084 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v"] Nov 04 01:27:53.725736 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:27:53.725705 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0edbd841_3fd2_4f5c_ad82_e28ab6dabe3b.slice/crio-bebe1de71b03c5f8888ab93997d1c3ed75ac90e3fb56a4df2157a3b637240eb2 WatchSource:0}: Error finding container bebe1de71b03c5f8888ab93997d1c3ed75ac90e3fb56a4df2157a3b637240eb2: Status 404 returned error can't find the container with id bebe1de71b03c5f8888ab93997d1c3ed75ac90e3fb56a4df2157a3b637240eb2 Nov 04 01:27:54.257661 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:54.257621 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" event={"ID":"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b","Type":"ContainerStarted","Data":"15502e66772ce886ef203af0b28ababf8f2156985e9d3027f24ff3a727f46b54"} Nov 04 01:27:54.258059 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:54.257674 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" event={"ID":"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b","Type":"ContainerStarted","Data":"bebe1de71b03c5f8888ab93997d1c3ed75ac90e3fb56a4df2157a3b637240eb2"} Nov 04 01:27:54.348175 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:54.348143 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:27:54.368057 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:54.368006 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" podStartSLOduration=2.367992352 podStartE2EDuration="2.367992352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:54.36720472 +0000 UTC m=+1024.398938108" watchObservedRunningTime="2026-11-04 01:27:54.367992352 +0000 UTC m=+1024.399725741" Nov 04 01:27:56.983308 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:27:56.983276 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:28:03.348481 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:03.348447 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:28:03.360549 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:03.360525 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:28:04.328169 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:04.328135 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:28:17.098684 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:17.098655 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:28:19.641515 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:28:19.641472 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:28:32.798327 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:32.798293 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:28:49.641664 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:28:49.641629 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:28:50.312339 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.312307 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:28:50.511187 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.511154 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv"] Nov 04 01:28:50.515010 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.514990 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.564364 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.564287 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a96a99\"/\"openshift-service-ca.crt\"" Nov 04 01:28:50.600178 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.600141 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-tmp-dir\") pod \"premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.600359 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.600194 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-home\") pod \"premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.600359 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.600331 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnks2\" (UniqueName: \"kubernetes.io/projected/dfd73b41-d89f-4391-a0aa-1246b07c718c-kube-api-access-mnks2\") pod \"premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.600507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.600493 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-model-cache\") pod \"premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.600559 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.600521 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfd73b41-d89f-4391-a0aa-1246b07c718c-tls-certs\") pod \"premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.600609 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.600558 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-dshm\") pod \"premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.610740 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.610716 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a96a99\"/\"kube-root-ca.crt\"" Nov 04 01:28:50.610884 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.610864 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a96a99\"/\"default-dockercfg-gkd6n\"" Nov 04 01:28:50.610884 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.610864 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a96a99\"/\"premium-w0-a96a99-kserve-self-signed-certs\"" Nov 04 01:28:50.613303 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.613284 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv"] Nov 04 01:28:50.701775 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.701742 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-dshm\") pod \"premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.702246 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.701811 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-tmp-dir\") pod \"premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.702246 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.701838 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-home\") pod \"premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.702246 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.701861 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnks2\" (UniqueName: \"kubernetes.io/projected/dfd73b41-d89f-4391-a0aa-1246b07c718c-kube-api-access-mnks2\") pod \"premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.702246 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.702006 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-model-cache\") pod \"premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.702246 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.702028 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfd73b41-d89f-4391-a0aa-1246b07c718c-tls-certs\") pod \"premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.702478 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.702252 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-tmp-dir\") pod \"premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.702478 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.702285 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-home\") pod \"premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.702478 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.702365 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-model-cache\") pod \"premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.704131 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.704110 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-dshm\") pod \"premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.704591 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.704572 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfd73b41-d89f-4391-a0aa-1246b07c718c-tls-certs\") pod \"premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.759160 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.759130 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnks2\" (UniqueName: \"kubernetes.io/projected/dfd73b41-d89f-4391-a0aa-1246b07c718c-kube-api-access-mnks2\") pod \"premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:50.827137 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:50.827052 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:28:51.199331 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:51.199294 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv"] Nov 04 01:28:51.205614 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:28:51.205585 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfd73b41_d89f_4391_a0aa_1246b07c718c.slice/crio-3ef2ac342122a6fc10a5d3b5ec732f9102c62c3909e3ebd56a5497d7fb91e6f5 WatchSource:0}: Error finding container 3ef2ac342122a6fc10a5d3b5ec732f9102c62c3909e3ebd56a5497d7fb91e6f5: Status 404 returned error can't find the container with id 3ef2ac342122a6fc10a5d3b5ec732f9102c62c3909e3ebd56a5497d7fb91e6f5 Nov 04 01:28:51.595243 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:51.595160 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" event={"ID":"dfd73b41-d89f-4391-a0aa-1246b07c718c","Type":"ContainerStarted","Data":"aa3c715a796d4afe82daea8432933ef20add3829b35b23701fea741ebce1790d"} Nov 04 01:28:51.595243 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:51.595200 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" event={"ID":"dfd73b41-d89f-4391-a0aa-1246b07c718c","Type":"ContainerStarted","Data":"3ef2ac342122a6fc10a5d3b5ec732f9102c62c3909e3ebd56a5497d7fb91e6f5"} Nov 04 01:28:51.707123 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:51.707074 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" podStartSLOduration=1.707059619 podStartE2EDuration="1.707059619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:51.705021741 +0000 UTC m=+1081.736755132" watchObservedRunningTime="2026-11-04 01:28:51.707059619 +0000 UTC m=+1081.738793008" Nov 04 01:28:51.827453 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:28:51.827311 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:29:00.828155 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:00.828071 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:29:00.840892 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:00.840869 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:29:01.679509 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:01.679480 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:29:07.066742 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:07.066707 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:29:19.641396 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:29:19.641358 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:29:23.634480 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:23.634410 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:29:27.423009 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.422970 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g"] Nov 04 01:29:27.427484 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.427459 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.474445 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.474398 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgxbj\" (UniqueName: \"kubernetes.io/projected/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-kube-api-access-kgxbj\") pod \"premium-w4-b2eadb-kserve-847675f59d-2nc5g\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.474597 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.474457 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-tmp-dir\") pod \"premium-w4-b2eadb-kserve-847675f59d-2nc5g\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.474597 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.474565 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-dshm\") pod \"premium-w4-b2eadb-kserve-847675f59d-2nc5g\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.474721 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.474616 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-home\") pod \"premium-w4-b2eadb-kserve-847675f59d-2nc5g\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.474721 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.474654 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-model-cache\") pod \"premium-w4-b2eadb-kserve-847675f59d-2nc5g\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.474816 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.474748 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-tls-certs\") pod \"premium-w4-b2eadb-kserve-847675f59d-2nc5g\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.523105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.523073 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b2eadb\"/\"openshift-service-ca.crt\"" Nov 04 01:29:27.523105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.523089 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b2eadb\"/\"premium-w4-b2eadb-kserve-self-signed-certs\"" Nov 04 01:29:27.523296 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.523113 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b2eadb\"/\"default-dockercfg-hzvpz\"" Nov 04 01:29:27.523296 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.523191 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b2eadb\"/\"kube-root-ca.crt\"" Nov 04 01:29:27.525712 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.525692 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g"] Nov 04 01:29:27.583029 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.580044 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgxbj\" (UniqueName: \"kubernetes.io/projected/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-kube-api-access-kgxbj\") pod \"premium-w4-b2eadb-kserve-847675f59d-2nc5g\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.583029 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.580111 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-tmp-dir\") pod \"premium-w4-b2eadb-kserve-847675f59d-2nc5g\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.583029 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.580145 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-dshm\") pod \"premium-w4-b2eadb-kserve-847675f59d-2nc5g\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.583029 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.580171 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-home\") pod \"premium-w4-b2eadb-kserve-847675f59d-2nc5g\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.583029 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.580210 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-model-cache\") pod \"premium-w4-b2eadb-kserve-847675f59d-2nc5g\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.583029 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.580265 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-tls-certs\") pod \"premium-w4-b2eadb-kserve-847675f59d-2nc5g\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.583584 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.583478 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-home\") pod \"premium-w4-b2eadb-kserve-847675f59d-2nc5g\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.583584 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.583576 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-model-cache\") pod \"premium-w4-b2eadb-kserve-847675f59d-2nc5g\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.583770 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.583728 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-tmp-dir\") pod \"premium-w4-b2eadb-kserve-847675f59d-2nc5g\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.589223 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.589192 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-tls-certs\") pod \"premium-w4-b2eadb-kserve-847675f59d-2nc5g\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.590035 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.590013 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-dshm\") pod \"premium-w4-b2eadb-kserve-847675f59d-2nc5g\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.640790 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.640762 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgxbj\" (UniqueName: \"kubernetes.io/projected/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-kube-api-access-kgxbj\") pod \"premium-w4-b2eadb-kserve-847675f59d-2nc5g\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:27.739627 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:27.739590 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:28.110475 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:28.110360 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g"] Nov 04 01:29:28.116515 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:29:28.116483 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f805d52_e8c2_4aa4_ab1c_3b52679eb1ee.slice/crio-c2d35ab2cb291d4b61eec050135c8680293cc18feebe1e115d05f7ef26709453 WatchSource:0}: Error finding container c2d35ab2cb291d4b61eec050135c8680293cc18feebe1e115d05f7ef26709453: Status 404 returned error can't find the container with id c2d35ab2cb291d4b61eec050135c8680293cc18feebe1e115d05f7ef26709453 Nov 04 01:29:28.849802 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:28.849760 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" event={"ID":"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee","Type":"ContainerStarted","Data":"c5e52b7aa75279cfd5443b22fdb3f23e42e27e54736d3bbf910cc1d90089f7d1"} Nov 04 01:29:28.849802 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:28.849804 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" event={"ID":"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee","Type":"ContainerStarted","Data":"c2d35ab2cb291d4b61eec050135c8680293cc18feebe1e115d05f7ef26709453"} Nov 04 01:29:28.961719 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:28.961665 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" podStartSLOduration=1.96164925 podStartE2EDuration="1.96164925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:28.95963921 +0000 UTC m=+1118.991372601" watchObservedRunningTime="2026-11-04 01:29:28.96164925 +0000 UTC m=+1118.993382643" Nov 04 01:29:29.740142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:29.740106 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:37.740503 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:37.740466 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:37.753234 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:37.753209 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:37.919445 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:37.918866 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:29:45.639388 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:45.639352 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:29:49.641106 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:29:49.641067 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:29:55.505605 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:55.505570 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8"] Nov 04 01:29:55.506156 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:29:55.505817 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" podUID="66684988-7433-4da1-94fb-8110cf7c31c7" containerName="istio-proxy" containerID="cri-o://268a54db80c8df84011c330a8101e71470887631d5bc801a20ef1d25b03d9c53" gracePeriod=30 Nov 04 01:30:00.168263 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.168227 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl"] Nov 04 01:30:00.172946 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.172919 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7"] Nov 04 01:30:00.173116 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.173069 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl" Nov 04 01:30:00.177328 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.177305 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl"] Nov 04 01:30:00.177466 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.177433 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7" Nov 04 01:30:00.181500 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.181479 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" Nov 04 01:30:00.214515 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.214490 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7"] Nov 04 01:30:00.218082 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.218062 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7" Nov 04 01:30:00.224576 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.224555 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:30:00.224710 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.224558 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:30:00.224710 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.224634 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w799x\"" Nov 04 01:30:00.269576 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.269500 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7"] Nov 04 01:30:00.270577 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.270555 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl"] Nov 04 01:30:00.281481 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.281454 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7"] Nov 04 01:30:00.324970 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.324933 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e84e7b4-91ce-406d-96d4-1aa1d6d2e597-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7\" (UID: \"8e84e7b4-91ce-406d-96d4-1aa1d6d2e597\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7" Nov 04 01:30:00.325143 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.325000 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp9kj\" (UniqueName: \"kubernetes.io/projected/0347a89c-4524-4b65-b29a-37a29adfff63-kube-api-access-cp9kj\") pod \"maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl\" (UID: \"0347a89c-4524-4b65-b29a-37a29adfff63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl" Nov 04 01:30:00.325214 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.325141 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zblmn\" (UniqueName: \"kubernetes.io/projected/8b69798c-73c6-4356-a065-e050b196cf14-kube-api-access-zblmn\") pod \"maas-api-key-cleanup-29895930-7h5p7\" (UID: \"8b69798c-73c6-4356-a065-e050b196cf14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7" Nov 04 01:30:00.325214 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.325177 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa1061e-0782-40db-96ce-0aee23fa54d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl\" (UID: \"daa1061e-0782-40db-96ce-0aee23fa54d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" Nov 04 01:30:00.325326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.325292 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b69798c-73c6-4356-a065-e050b196cf14-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895930-7h5p7\" (UID: \"8b69798c-73c6-4356-a065-e050b196cf14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7" Nov 04 01:30:00.325378 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.325355 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr7nc\" (UniqueName: \"kubernetes.io/projected/8e84e7b4-91ce-406d-96d4-1aa1d6d2e597-kube-api-access-wr7nc\") pod \"maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7\" (UID: \"8e84e7b4-91ce-406d-96d4-1aa1d6d2e597\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7" Nov 04 01:30:00.325457 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.325439 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlztk\" (UniqueName: \"kubernetes.io/projected/daa1061e-0782-40db-96ce-0aee23fa54d4-kube-api-access-jlztk\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl\" (UID: \"daa1061e-0782-40db-96ce-0aee23fa54d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" Nov 04 01:30:00.325515 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.325487 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0347a89c-4524-4b65-b29a-37a29adfff63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl\" (UID: \"0347a89c-4524-4b65-b29a-37a29adfff63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl" Nov 04 01:30:00.333062 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.333037 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl"] Nov 04 01:30:00.426510 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.426403 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlztk\" (UniqueName: \"kubernetes.io/projected/daa1061e-0782-40db-96ce-0aee23fa54d4-kube-api-access-jlztk\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl\" (UID: \"daa1061e-0782-40db-96ce-0aee23fa54d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" Nov 04 01:30:00.426656 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.426545 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0347a89c-4524-4b65-b29a-37a29adfff63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl\" (UID: \"0347a89c-4524-4b65-b29a-37a29adfff63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl" Nov 04 01:30:00.426656 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.426564 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e84e7b4-91ce-406d-96d4-1aa1d6d2e597-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7\" (UID: \"8e84e7b4-91ce-406d-96d4-1aa1d6d2e597\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7" Nov 04 01:30:00.426656 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.426596 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp9kj\" (UniqueName: \"kubernetes.io/projected/0347a89c-4524-4b65-b29a-37a29adfff63-kube-api-access-cp9kj\") pod \"maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl\" (UID: \"0347a89c-4524-4b65-b29a-37a29adfff63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl" Nov 04 01:30:00.426757 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.426734 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zblmn\" (UniqueName: \"kubernetes.io/projected/8b69798c-73c6-4356-a065-e050b196cf14-kube-api-access-zblmn\") pod \"maas-api-key-cleanup-29895930-7h5p7\" (UID: \"8b69798c-73c6-4356-a065-e050b196cf14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7" Nov 04 01:30:00.426790 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.426773 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa1061e-0782-40db-96ce-0aee23fa54d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl\" (UID: \"daa1061e-0782-40db-96ce-0aee23fa54d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" Nov 04 01:30:00.426880 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.426864 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b69798c-73c6-4356-a065-e050b196cf14-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895930-7h5p7\" (UID: \"8b69798c-73c6-4356-a065-e050b196cf14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7" Nov 04 01:30:00.426945 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.426928 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr7nc\" (UniqueName: \"kubernetes.io/projected/8e84e7b4-91ce-406d-96d4-1aa1d6d2e597-kube-api-access-wr7nc\") pod \"maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7\" (UID: \"8e84e7b4-91ce-406d-96d4-1aa1d6d2e597\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7" Nov 04 01:30:00.427310 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.427289 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0347a89c-4524-4b65-b29a-37a29adfff63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl\" (UID: \"0347a89c-4524-4b65-b29a-37a29adfff63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl" Nov 04 01:30:00.427408 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.427392 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e84e7b4-91ce-406d-96d4-1aa1d6d2e597-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7\" (UID: \"8e84e7b4-91ce-406d-96d4-1aa1d6d2e597\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7" Nov 04 01:30:00.427480 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.427403 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa1061e-0782-40db-96ce-0aee23fa54d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl\" (UID: \"daa1061e-0782-40db-96ce-0aee23fa54d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" Nov 04 01:30:00.427558 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.427541 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b69798c-73c6-4356-a065-e050b196cf14-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895930-7h5p7\" (UID: \"8b69798c-73c6-4356-a065-e050b196cf14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7" Nov 04 01:30:00.484839 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.484811 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zblmn\" (UniqueName: \"kubernetes.io/projected/8b69798c-73c6-4356-a065-e050b196cf14-kube-api-access-zblmn\") pod \"maas-api-key-cleanup-29895930-7h5p7\" (UID: \"8b69798c-73c6-4356-a065-e050b196cf14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7" Nov 04 01:30:00.485018 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.484897 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr7nc\" (UniqueName: \"kubernetes.io/projected/8e84e7b4-91ce-406d-96d4-1aa1d6d2e597-kube-api-access-wr7nc\") pod \"maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7\" (UID: \"8e84e7b4-91ce-406d-96d4-1aa1d6d2e597\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7" Nov 04 01:30:00.485018 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.484913 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlztk\" (UniqueName: \"kubernetes.io/projected/daa1061e-0782-40db-96ce-0aee23fa54d4-kube-api-access-jlztk\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl\" (UID: \"daa1061e-0782-40db-96ce-0aee23fa54d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" Nov 04 01:30:00.485161 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.485143 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp9kj\" (UniqueName: \"kubernetes.io/projected/0347a89c-4524-4b65-b29a-37a29adfff63-kube-api-access-cp9kj\") pod \"maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl\" (UID: \"0347a89c-4524-4b65-b29a-37a29adfff63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl" Nov 04 01:30:00.494299 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.494276 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl" Nov 04 01:30:00.503313 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.503292 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7" Nov 04 01:30:00.510814 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.510791 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" Nov 04 01:30:00.530034 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.530008 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7" Nov 04 01:30:00.840975 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.840952 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:30:00.906304 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.906265 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl"] Nov 04 01:30:00.912769 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:30:00.912741 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa1061e_0782_40db_96ce_0aee23fa54d4.slice/crio-7f5f31879375987cfefe98a9f088a57b6bf610264e23886978f3d5a2265ddb23 WatchSource:0}: Error finding container 7f5f31879375987cfefe98a9f088a57b6bf610264e23886978f3d5a2265ddb23: Status 404 returned error can't find the container with id 7f5f31879375987cfefe98a9f088a57b6bf610264e23886978f3d5a2265ddb23 Nov 04 01:30:00.914645 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.914624 5995 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:30:00.955980 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.955952 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl"] Nov 04 01:30:00.959882 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:00.959857 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7"] Nov 04 01:30:01.004184 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.004154 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7"] Nov 04 01:30:01.032612 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.032586 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-workload-certs\") pod \"66684988-7433-4da1-94fb-8110cf7c31c7\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " Nov 04 01:30:01.032719 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.032651 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-istio-envoy\") pod \"66684988-7433-4da1-94fb-8110cf7c31c7\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " Nov 04 01:30:01.032719 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.032683 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-istio-data\") pod \"66684988-7433-4da1-94fb-8110cf7c31c7\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " Nov 04 01:30:01.032719 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.032704 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66684988-7433-4da1-94fb-8110cf7c31c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66684988-7433-4da1-94fb-8110cf7c31c7-istio-podinfo\") pod \"66684988-7433-4da1-94fb-8110cf7c31c7\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " Nov 04 01:30:01.032855 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.032729 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66684988-7433-4da1-94fb-8110cf7c31c7-kube-api-access-5h76w\" (UniqueName: \"kubernetes.io/projected/66684988-7433-4da1-94fb-8110cf7c31c7-kube-api-access-5h76w\") pod \"66684988-7433-4da1-94fb-8110cf7c31c7\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " Nov 04 01:30:01.032855 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.032747 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66684988-7433-4da1-94fb-8110cf7c31c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66684988-7433-4da1-94fb-8110cf7c31c7-istiod-ca-cert\") pod \"66684988-7433-4da1-94fb-8110cf7c31c7\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " Nov 04 01:30:01.032855 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.032792 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66684988-7433-4da1-94fb-8110cf7c31c7-istio-token\" (UniqueName: \"kubernetes.io/projected/66684988-7433-4da1-94fb-8110cf7c31c7-istio-token\") pod \"66684988-7433-4da1-94fb-8110cf7c31c7\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " Nov 04 01:30:01.033000 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.032885 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-credential-socket\") pod \"66684988-7433-4da1-94fb-8110cf7c31c7\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " Nov 04 01:30:01.033000 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.032902 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-workload-certs" pod "66684988-7433-4da1-94fb-8110cf7c31c7" (UID: "66684988-7433-4da1-94fb-8110cf7c31c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:01.033000 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.032920 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-istio-data" pod "66684988-7433-4da1-94fb-8110cf7c31c7" (UID: "66684988-7433-4da1-94fb-8110cf7c31c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:01.033000 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.032926 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-workload-socket\") pod \"66684988-7433-4da1-94fb-8110cf7c31c7\" (UID: \"66684988-7433-4da1-94fb-8110cf7c31c7\") " Nov 04 01:30:01.033196 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.033124 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-workload-socket" pod "66684988-7433-4da1-94fb-8110cf7c31c7" (UID: "66684988-7433-4da1-94fb-8110cf7c31c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:01.033249 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.033196 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66684988-7433-4da1-94fb-8110cf7c31c7-istiod-ca-cert" pod "66684988-7433-4da1-94fb-8110cf7c31c7" (UID: "66684988-7433-4da1-94fb-8110cf7c31c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:01.033354 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.033333 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-credential-socket" pod "66684988-7433-4da1-94fb-8110cf7c31c7" (UID: "66684988-7433-4da1-94fb-8110cf7c31c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:01.033588 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.033566 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:01.033588 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.033589 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:01.033765 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.033604 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66684988-7433-4da1-94fb-8110cf7c31c7-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:01.033765 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.033619 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:01.033765 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.033633 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:01.035018 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.034993 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66684988-7433-4da1-94fb-8110cf7c31c7-istio-token" pod "66684988-7433-4da1-94fb-8110cf7c31c7" (UID: "66684988-7433-4da1-94fb-8110cf7c31c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:01.035091 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.035019 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66684988-7433-4da1-94fb-8110cf7c31c7-istio-podinfo" pod "66684988-7433-4da1-94fb-8110cf7c31c7" (UID: "66684988-7433-4da1-94fb-8110cf7c31c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:01.035091 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.035067 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-istio-envoy" pod "66684988-7433-4da1-94fb-8110cf7c31c7" (UID: "66684988-7433-4da1-94fb-8110cf7c31c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:01.035091 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.035068 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66684988-7433-4da1-94fb-8110cf7c31c7-kube-api-access-5h76w" pod "66684988-7433-4da1-94fb-8110cf7c31c7" (UID: "66684988-7433-4da1-94fb-8110cf7c31c7"). InnerVolumeSpecName "kube-api-access-5h76w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:01.051483 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.051456 5995 generic.go:356] "Generic (PLEG): container finished" podID="66684988-7433-4da1-94fb-8110cf7c31c7" containerID="268a54db80c8df84011c330a8101e71470887631d5bc801a20ef1d25b03d9c53" exitCode=0 Nov 04 01:30:01.051582 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.051505 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" Nov 04 01:30:01.051582 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.051537 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" event={"ID":"66684988-7433-4da1-94fb-8110cf7c31c7","Type":"ContainerDied","Data":"268a54db80c8df84011c330a8101e71470887631d5bc801a20ef1d25b03d9c53"} Nov 04 01:30:01.051691 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.051595 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8" event={"ID":"66684988-7433-4da1-94fb-8110cf7c31c7","Type":"ContainerDied","Data":"36ff9035c24e9b2413f3b8eacc65e46ac9e1f1b933349f104fbb7b16920b0150"} Nov 04 01:30:01.051691 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.051624 5995 scope.go:122] "RemoveContainer" containerID="268a54db80c8df84011c330a8101e71470887631d5bc801a20ef1d25b03d9c53" Nov 04 01:30:01.052801 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.052781 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" event={"ID":"daa1061e-0782-40db-96ce-0aee23fa54d4","Type":"ContainerStarted","Data":"7f5f31879375987cfefe98a9f088a57b6bf610264e23886978f3d5a2265ddb23"} Nov 04 01:30:01.054246 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.054223 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7" event={"ID":"8e84e7b4-91ce-406d-96d4-1aa1d6d2e597","Type":"ContainerStarted","Data":"20c719eba890d26bc68b912b806f6ffc92e334ffb19229c59e59ac16229e7998"} Nov 04 01:30:01.055567 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.055545 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7" event={"ID":"8b69798c-73c6-4356-a065-e050b196cf14","Type":"ContainerStarted","Data":"f76a393073ff6869f3d45b6930648c2a90b23899a6ab5006cde443b8c2b38475"} Nov 04 01:30:01.056788 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.056768 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl" event={"ID":"0347a89c-4524-4b65-b29a-37a29adfff63","Type":"ContainerStarted","Data":"053f6094a754bde18821cf2a02a3349ea6c368d0b3b6e7d5474f83a4c38aa837"} Nov 04 01:30:01.061350 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.061336 5995 scope.go:122] "RemoveContainer" containerID="268a54db80c8df84011c330a8101e71470887631d5bc801a20ef1d25b03d9c53" Nov 04 01:30:01.061630 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:30:01.061611 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"268a54db80c8df84011c330a8101e71470887631d5bc801a20ef1d25b03d9c53\": container with ID starting with 268a54db80c8df84011c330a8101e71470887631d5bc801a20ef1d25b03d9c53 not found: ID does not exist" containerID="268a54db80c8df84011c330a8101e71470887631d5bc801a20ef1d25b03d9c53" Nov 04 01:30:01.061708 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.061637 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"268a54db80c8df84011c330a8101e71470887631d5bc801a20ef1d25b03d9c53"} err="failed to get container status \"268a54db80c8df84011c330a8101e71470887631d5bc801a20ef1d25b03d9c53\": rpc error: code = NotFound desc = could not find container \"268a54db80c8df84011c330a8101e71470887631d5bc801a20ef1d25b03d9c53\": container with ID starting with 268a54db80c8df84011c330a8101e71470887631d5bc801a20ef1d25b03d9c53 not found: ID does not exist" Nov 04 01:30:01.134511 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.134476 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66684988-7433-4da1-94fb-8110cf7c31c7-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:01.134511 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.134502 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66684988-7433-4da1-94fb-8110cf7c31c7-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:01.134511 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.134513 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h76w\" (UniqueName: \"kubernetes.io/projected/66684988-7433-4da1-94fb-8110cf7c31c7-kube-api-access-5h76w\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:01.134730 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.134522 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66684988-7433-4da1-94fb-8110cf7c31c7-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:01.206831 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.206737 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8"] Nov 04 01:30:01.210924 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:01.210897 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7w5l8"] Nov 04 01:30:02.132354 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:02.132318 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66684988-7433-4da1-94fb-8110cf7c31c7" path="/var/lib/kubelet/pods/66684988-7433-4da1-94fb-8110cf7c31c7/volumes" Nov 04 01:30:06.097247 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:06.097220 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" event={"ID":"daa1061e-0782-40db-96ce-0aee23fa54d4","Type":"ContainerStarted","Data":"e658c2ed8d9550a81c5441b95ec1be1c1cccaba1b078d85a795b3c0651d382d5"} Nov 04 01:30:06.206596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:06.206555 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" podStartSLOduration=1.479387836 podStartE2EDuration="6.206542938s" totalImagesPullingTime="4.727155102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:06.206307084 +0000 UTC m=+1156.238040474" watchObservedRunningTime="2026-11-04 01:30:06.206542938 +0000 UTC m=+1156.238276351" Nov 04 01:30:07.105423 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:07.105390 5995 generic.go:356] "Generic (PLEG): container finished" podID="0347a89c-4524-4b65-b29a-37a29adfff63" containerID="2eeb210db2df91ffe87d967f86096e491ece8bb652d69a71ccde92cb220e32e3" exitCode=60 Nov 04 01:30:07.105896 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:07.105482 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl" event={"ID":"0347a89c-4524-4b65-b29a-37a29adfff63","Type":"ContainerDied","Data":"2eeb210db2df91ffe87d967f86096e491ece8bb652d69a71ccde92cb220e32e3"} Nov 04 01:30:07.105896 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:07.105716 5995 scope.go:122] "RemoveContainer" containerID="2eeb210db2df91ffe87d967f86096e491ece8bb652d69a71ccde92cb220e32e3" Nov 04 01:30:07.107350 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:07.107329 5995 generic.go:356] "Generic (PLEG): container finished" podID="daa1061e-0782-40db-96ce-0aee23fa54d4" containerID="e658c2ed8d9550a81c5441b95ec1be1c1cccaba1b078d85a795b3c0651d382d5" exitCode=60 Nov 04 01:30:07.107497 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:07.107448 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" event={"ID":"daa1061e-0782-40db-96ce-0aee23fa54d4","Type":"ContainerDied","Data":"e658c2ed8d9550a81c5441b95ec1be1c1cccaba1b078d85a795b3c0651d382d5"} Nov 04 01:30:07.107663 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:07.107642 5995 scope.go:122] "RemoveContainer" containerID="e658c2ed8d9550a81c5441b95ec1be1c1cccaba1b078d85a795b3c0651d382d5" Nov 04 01:30:07.109726 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:07.109705 5995 generic.go:356] "Generic (PLEG): container finished" podID="8e84e7b4-91ce-406d-96d4-1aa1d6d2e597" containerID="97816fe28431c4a6235ebc8f09494c3b936077fdb8e9ed1f20d41d2def781715" exitCode=60 Nov 04 01:30:07.109816 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:07.109792 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7" event={"ID":"8e84e7b4-91ce-406d-96d4-1aa1d6d2e597","Type":"ContainerDied","Data":"97816fe28431c4a6235ebc8f09494c3b936077fdb8e9ed1f20d41d2def781715"} Nov 04 01:30:07.109964 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:07.109949 5995 scope.go:122] "RemoveContainer" containerID="97816fe28431c4a6235ebc8f09494c3b936077fdb8e9ed1f20d41d2def781715" Nov 04 01:30:07.112536 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:07.112489 5995 generic.go:356] "Generic (PLEG): container finished" podID="8b69798c-73c6-4356-a065-e050b196cf14" containerID="72be2d4da6a12cfac06b8210586965e600bf4b9b8415ce45858a18f0dc4b6e20" exitCode=60 Nov 04 01:30:07.112610 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:07.112539 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7" event={"ID":"8b69798c-73c6-4356-a065-e050b196cf14","Type":"ContainerDied","Data":"72be2d4da6a12cfac06b8210586965e600bf4b9b8415ce45858a18f0dc4b6e20"} Nov 04 01:30:07.112721 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:07.112710 5995 scope.go:122] "RemoveContainer" containerID="72be2d4da6a12cfac06b8210586965e600bf4b9b8415ce45858a18f0dc4b6e20" Nov 04 01:30:08.120203 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:08.120173 5995 generic.go:356] "Generic (PLEG): container finished" podID="daa1061e-0782-40db-96ce-0aee23fa54d4" containerID="779e2f8fbf545b1294fdb53938ec2997e7a8f4d5feb5ec8071a612fa2c13dbe5" exitCode=60 Nov 04 01:30:08.120627 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:08.120259 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" event={"ID":"daa1061e-0782-40db-96ce-0aee23fa54d4","Type":"ContainerDied","Data":"779e2f8fbf545b1294fdb53938ec2997e7a8f4d5feb5ec8071a612fa2c13dbe5"} Nov 04 01:30:08.120627 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:08.120333 5995 scope.go:122] "RemoveContainer" containerID="e658c2ed8d9550a81c5441b95ec1be1c1cccaba1b078d85a795b3c0651d382d5" Nov 04 01:30:08.120627 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:08.120518 5995 scope.go:122] "RemoveContainer" containerID="779e2f8fbf545b1294fdb53938ec2997e7a8f4d5feb5ec8071a612fa2c13dbe5" Nov 04 01:30:08.120778 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:30:08.120764 5995 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl_odh-ai-gateway-infra(daa1061e-0782-40db-96ce-0aee23fa54d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" podUID="daa1061e-0782-40db-96ce-0aee23fa54d4" Nov 04 01:30:08.122339 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:08.122317 5995 generic.go:356] "Generic (PLEG): container finished" podID="8e84e7b4-91ce-406d-96d4-1aa1d6d2e597" containerID="ca3b49b4b3fbb97df4003946f7dbad05e7f7f9036d624ee52062fa2b8c0abba5" exitCode=60 Nov 04 01:30:08.122665 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:08.122649 5995 scope.go:122] "RemoveContainer" containerID="ca3b49b4b3fbb97df4003946f7dbad05e7f7f9036d624ee52062fa2b8c0abba5" Nov 04 01:30:08.122885 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:30:08.122866 5995 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7_odh-ai-gateway-infra(8e84e7b4-91ce-406d-96d4-1aa1d6d2e597)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7" podUID="8e84e7b4-91ce-406d-96d4-1aa1d6d2e597" Nov 04 01:30:08.124244 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:08.124226 5995 generic.go:356] "Generic (PLEG): container finished" podID="8b69798c-73c6-4356-a065-e050b196cf14" containerID="0c711a7f4cd7ceebc06df8a82d8b93e085056e164f79ec48874c058ad26b5bcb" exitCode=60 Nov 04 01:30:08.124560 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:08.124546 5995 scope.go:122] "RemoveContainer" containerID="0c711a7f4cd7ceebc06df8a82d8b93e085056e164f79ec48874c058ad26b5bcb" Nov 04 01:30:08.124749 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:30:08.124733 5995 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895930-7h5p7_odh-ai-gateway-infra(8b69798c-73c6-4356-a065-e050b196cf14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7" podUID="8b69798c-73c6-4356-a065-e050b196cf14" Nov 04 01:30:08.126376 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:08.126355 5995 generic.go:356] "Generic (PLEG): container finished" podID="0347a89c-4524-4b65-b29a-37a29adfff63" containerID="30a31802cc611e7e59ba7a0cb4d79b90be75a1cbdf74d643117a949436e924b7" exitCode=60 Nov 04 01:30:08.126700 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:08.126681 5995 scope.go:122] "RemoveContainer" containerID="30a31802cc611e7e59ba7a0cb4d79b90be75a1cbdf74d643117a949436e924b7" Nov 04 01:30:08.126929 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:30:08.126909 5995 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl_odh-ai-gateway-infra(0347a89c-4524-4b65-b29a-37a29adfff63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl" podUID="0347a89c-4524-4b65-b29a-37a29adfff63" Nov 04 01:30:08.129020 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:08.128995 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7" event={"ID":"8e84e7b4-91ce-406d-96d4-1aa1d6d2e597","Type":"ContainerDied","Data":"ca3b49b4b3fbb97df4003946f7dbad05e7f7f9036d624ee52062fa2b8c0abba5"} Nov 04 01:30:08.129117 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:08.129046 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7" event={"ID":"8b69798c-73c6-4356-a065-e050b196cf14","Type":"ContainerDied","Data":"0c711a7f4cd7ceebc06df8a82d8b93e085056e164f79ec48874c058ad26b5bcb"} Nov 04 01:30:08.129117 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:08.129075 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl" event={"ID":"0347a89c-4524-4b65-b29a-37a29adfff63","Type":"ContainerDied","Data":"30a31802cc611e7e59ba7a0cb4d79b90be75a1cbdf74d643117a949436e924b7"} Nov 04 01:30:08.268630 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:08.268577 5995 scope.go:122] "RemoveContainer" containerID="97816fe28431c4a6235ebc8f09494c3b936077fdb8e9ed1f20d41d2def781715" Nov 04 01:30:08.282272 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:08.282243 5995 scope.go:122] "RemoveContainer" containerID="72be2d4da6a12cfac06b8210586965e600bf4b9b8415ce45858a18f0dc4b6e20" Nov 04 01:30:08.294385 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:08.294360 5995 scope.go:122] "RemoveContainer" containerID="2eeb210db2df91ffe87d967f86096e491ece8bb652d69a71ccde92cb220e32e3" Nov 04 01:30:09.133578 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:09.133551 5995 scope.go:122] "RemoveContainer" containerID="30a31802cc611e7e59ba7a0cb4d79b90be75a1cbdf74d643117a949436e924b7" Nov 04 01:30:09.134005 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:30:09.133838 5995 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl_odh-ai-gateway-infra(0347a89c-4524-4b65-b29a-37a29adfff63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl" podUID="0347a89c-4524-4b65-b29a-37a29adfff63" Nov 04 01:30:09.135126 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:09.135099 5995 scope.go:122] "RemoveContainer" containerID="779e2f8fbf545b1294fdb53938ec2997e7a8f4d5feb5ec8071a612fa2c13dbe5" Nov 04 01:30:09.135360 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:30:09.135313 5995 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl_odh-ai-gateway-infra(daa1061e-0782-40db-96ce-0aee23fa54d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" podUID="daa1061e-0782-40db-96ce-0aee23fa54d4" Nov 04 01:30:09.136616 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:09.136596 5995 scope.go:122] "RemoveContainer" containerID="ca3b49b4b3fbb97df4003946f7dbad05e7f7f9036d624ee52062fa2b8c0abba5" Nov 04 01:30:09.136807 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:30:09.136789 5995 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7_odh-ai-gateway-infra(8e84e7b4-91ce-406d-96d4-1aa1d6d2e597)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7" podUID="8e84e7b4-91ce-406d-96d4-1aa1d6d2e597" Nov 04 01:30:09.137952 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:09.137936 5995 scope.go:122] "RemoveContainer" containerID="0c711a7f4cd7ceebc06df8a82d8b93e085056e164f79ec48874c058ad26b5bcb" Nov 04 01:30:09.138112 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:30:09.138097 5995 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895930-7h5p7_odh-ai-gateway-infra(8b69798c-73c6-4356-a065-e050b196cf14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7" podUID="8b69798c-73c6-4356-a065-e050b196cf14" Nov 04 01:30:14.797838 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:14.797802 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:30:19.122469 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:19.122436 5995 scope.go:122] "RemoveContainer" containerID="0c711a7f4cd7ceebc06df8a82d8b93e085056e164f79ec48874c058ad26b5bcb" Nov 04 01:30:19.642272 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:30:19.642182 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:30:20.126013 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:20.125980 5995 scope.go:122] "RemoveContainer" containerID="ca3b49b4b3fbb97df4003946f7dbad05e7f7f9036d624ee52062fa2b8c0abba5" Nov 04 01:30:20.209321 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:20.209285 5995 generic.go:356] "Generic (PLEG): container finished" podID="8b69798c-73c6-4356-a065-e050b196cf14" containerID="0fd63b2c17215d6f31e23b20692cc8284b7ed8aebc4cdbab93168be3c108946d" exitCode=60 Nov 04 01:30:20.209508 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:20.209358 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7" event={"ID":"8b69798c-73c6-4356-a065-e050b196cf14","Type":"ContainerDied","Data":"0fd63b2c17215d6f31e23b20692cc8284b7ed8aebc4cdbab93168be3c108946d"} Nov 04 01:30:20.209508 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:20.209437 5995 scope.go:122] "RemoveContainer" containerID="0c711a7f4cd7ceebc06df8a82d8b93e085056e164f79ec48874c058ad26b5bcb" Nov 04 01:30:20.209792 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:20.209777 5995 scope.go:122] "RemoveContainer" containerID="0fd63b2c17215d6f31e23b20692cc8284b7ed8aebc4cdbab93168be3c108946d" Nov 04 01:30:20.210069 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:30:20.210049 5995 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895930-7h5p7_odh-ai-gateway-infra(8b69798c-73c6-4356-a065-e050b196cf14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7" podUID="8b69798c-73c6-4356-a065-e050b196cf14" Nov 04 01:30:21.122625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:21.122599 5995 scope.go:122] "RemoveContainer" containerID="779e2f8fbf545b1294fdb53938ec2997e7a8f4d5feb5ec8071a612fa2c13dbe5" Nov 04 01:30:21.122820 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:21.122786 5995 scope.go:122] "RemoveContainer" containerID="30a31802cc611e7e59ba7a0cb4d79b90be75a1cbdf74d643117a949436e924b7" Nov 04 01:30:21.217093 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:21.217023 5995 generic.go:356] "Generic (PLEG): container finished" podID="8e84e7b4-91ce-406d-96d4-1aa1d6d2e597" containerID="944fdaab5377c71abca206b20991e463810b3190293d7d8eb723834629a870b0" exitCode=60 Nov 04 01:30:21.217495 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:21.217088 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7" event={"ID":"8e84e7b4-91ce-406d-96d4-1aa1d6d2e597","Type":"ContainerDied","Data":"944fdaab5377c71abca206b20991e463810b3190293d7d8eb723834629a870b0"} Nov 04 01:30:21.217495 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:21.217155 5995 scope.go:122] "RemoveContainer" containerID="ca3b49b4b3fbb97df4003946f7dbad05e7f7f9036d624ee52062fa2b8c0abba5" Nov 04 01:30:21.217621 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:21.217536 5995 scope.go:122] "RemoveContainer" containerID="944fdaab5377c71abca206b20991e463810b3190293d7d8eb723834629a870b0" Nov 04 01:30:21.217796 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:30:21.217774 5995 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7_odh-ai-gateway-infra(8e84e7b4-91ce-406d-96d4-1aa1d6d2e597)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7" podUID="8e84e7b4-91ce-406d-96d4-1aa1d6d2e597" Nov 04 01:30:21.329936 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:21.329898 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7"] Nov 04 01:30:21.492251 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:21.492220 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7" Nov 04 01:30:21.550078 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:21.550050 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b69798c-73c6-4356-a065-e050b196cf14-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b69798c-73c6-4356-a065-e050b196cf14-maas-api-service-ca\") pod \"8b69798c-73c6-4356-a065-e050b196cf14\" (UID: \"8b69798c-73c6-4356-a065-e050b196cf14\") " Nov 04 01:30:21.550247 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:21.550105 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b69798c-73c6-4356-a065-e050b196cf14-kube-api-access-zblmn\" (UniqueName: \"kubernetes.io/projected/8b69798c-73c6-4356-a065-e050b196cf14-kube-api-access-zblmn\") pod \"8b69798c-73c6-4356-a065-e050b196cf14\" (UID: \"8b69798c-73c6-4356-a065-e050b196cf14\") " Nov 04 01:30:21.550389 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:21.550369 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b69798c-73c6-4356-a065-e050b196cf14-maas-api-service-ca" pod "8b69798c-73c6-4356-a065-e050b196cf14" (UID: "8b69798c-73c6-4356-a065-e050b196cf14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:21.550548 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:21.550531 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b69798c-73c6-4356-a065-e050b196cf14-maas-api-service-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:21.552309 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:21.552281 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b69798c-73c6-4356-a065-e050b196cf14-kube-api-access-zblmn" pod "8b69798c-73c6-4356-a065-e050b196cf14" (UID: "8b69798c-73c6-4356-a065-e050b196cf14"). InnerVolumeSpecName "kube-api-access-zblmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:21.651735 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:21.651703 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zblmn\" (UniqueName: \"kubernetes.io/projected/8b69798c-73c6-4356-a065-e050b196cf14-kube-api-access-zblmn\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:22.227404 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.227324 5995 generic.go:356] "Generic (PLEG): container finished" podID="0347a89c-4524-4b65-b29a-37a29adfff63" containerID="97b7928288e9dfd0ca676c2469bf581861cc42255fcd9d52282abb02184b77c0" exitCode=60 Nov 04 01:30:22.227849 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.227395 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl" event={"ID":"0347a89c-4524-4b65-b29a-37a29adfff63","Type":"ContainerDied","Data":"97b7928288e9dfd0ca676c2469bf581861cc42255fcd9d52282abb02184b77c0"} Nov 04 01:30:22.227849 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.227462 5995 scope.go:122] "RemoveContainer" containerID="30a31802cc611e7e59ba7a0cb4d79b90be75a1cbdf74d643117a949436e924b7" Nov 04 01:30:22.227986 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.227877 5995 scope.go:122] "RemoveContainer" containerID="97b7928288e9dfd0ca676c2469bf581861cc42255fcd9d52282abb02184b77c0" Nov 04 01:30:22.228182 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:30:22.228159 5995 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl_odh-ai-gateway-infra(0347a89c-4524-4b65-b29a-37a29adfff63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl" podUID="0347a89c-4524-4b65-b29a-37a29adfff63" Nov 04 01:30:22.229610 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.229588 5995 generic.go:356] "Generic (PLEG): container finished" podID="daa1061e-0782-40db-96ce-0aee23fa54d4" containerID="09f2d0069f1cc4a7dbf87a61a0aceb0c7395b3555c52049777992bf9ad3555d8" exitCode=60 Nov 04 01:30:22.229726 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.229665 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" event={"ID":"daa1061e-0782-40db-96ce-0aee23fa54d4","Type":"ContainerDied","Data":"09f2d0069f1cc4a7dbf87a61a0aceb0c7395b3555c52049777992bf9ad3555d8"} Nov 04 01:30:22.230086 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.230065 5995 scope.go:122] "RemoveContainer" containerID="09f2d0069f1cc4a7dbf87a61a0aceb0c7395b3555c52049777992bf9ad3555d8" Nov 04 01:30:22.230350 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:30:22.230326 5995 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl_odh-ai-gateway-infra(daa1061e-0782-40db-96ce-0aee23fa54d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" podUID="daa1061e-0782-40db-96ce-0aee23fa54d4" Nov 04 01:30:22.232624 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.232609 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7" Nov 04 01:30:22.232721 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.232652 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7" event={"ID":"8b69798c-73c6-4356-a065-e050b196cf14","Type":"ContainerDied","Data":"f76a393073ff6869f3d45b6930648c2a90b23899a6ab5006cde443b8c2b38475"} Nov 04 01:30:22.239794 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.239767 5995 scope.go:122] "RemoveContainer" containerID="779e2f8fbf545b1294fdb53938ec2997e7a8f4d5feb5ec8071a612fa2c13dbe5" Nov 04 01:30:22.251976 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.251950 5995 scope.go:122] "RemoveContainer" containerID="0fd63b2c17215d6f31e23b20692cc8284b7ed8aebc4cdbab93168be3c108946d" Nov 04 01:30:22.381295 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.381256 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7"] Nov 04 01:30:22.532617 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.532592 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7" Nov 04 01:30:22.553760 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.553722 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7"] Nov 04 01:30:22.560326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.560299 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-7h5p7"] Nov 04 01:30:22.661799 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.661765 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e84e7b4-91ce-406d-96d4-1aa1d6d2e597-kube-api-access-wr7nc\" (UniqueName: \"kubernetes.io/projected/8e84e7b4-91ce-406d-96d4-1aa1d6d2e597-kube-api-access-wr7nc\") pod \"8e84e7b4-91ce-406d-96d4-1aa1d6d2e597\" (UID: \"8e84e7b4-91ce-406d-96d4-1aa1d6d2e597\") " Nov 04 01:30:22.661994 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.661854 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e84e7b4-91ce-406d-96d4-1aa1d6d2e597-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e84e7b4-91ce-406d-96d4-1aa1d6d2e597-maas-api-service-ca\") pod \"8e84e7b4-91ce-406d-96d4-1aa1d6d2e597\" (UID: \"8e84e7b4-91ce-406d-96d4-1aa1d6d2e597\") " Nov 04 01:30:22.662255 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.662224 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e84e7b4-91ce-406d-96d4-1aa1d6d2e597-maas-api-service-ca" pod "8e84e7b4-91ce-406d-96d4-1aa1d6d2e597" (UID: "8e84e7b4-91ce-406d-96d4-1aa1d6d2e597"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:22.663998 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.663973 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e84e7b4-91ce-406d-96d4-1aa1d6d2e597-kube-api-access-wr7nc" pod "8e84e7b4-91ce-406d-96d4-1aa1d6d2e597" (UID: "8e84e7b4-91ce-406d-96d4-1aa1d6d2e597"). InnerVolumeSpecName "kube-api-access-wr7nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:22.763532 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.763449 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr7nc\" (UniqueName: \"kubernetes.io/projected/8e84e7b4-91ce-406d-96d4-1aa1d6d2e597-kube-api-access-wr7nc\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:22.763532 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:22.763485 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e84e7b4-91ce-406d-96d4-1aa1d6d2e597-maas-api-service-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:23.243946 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.243919 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7" Nov 04 01:30:23.243946 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.243922 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7" event={"ID":"8e84e7b4-91ce-406d-96d4-1aa1d6d2e597","Type":"ContainerDied","Data":"20c719eba890d26bc68b912b806f6ffc92e334ffb19229c59e59ac16229e7998"} Nov 04 01:30:23.244408 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.243981 5995 scope.go:122] "RemoveContainer" containerID="944fdaab5377c71abca206b20991e463810b3190293d7d8eb723834629a870b0" Nov 04 01:30:23.348175 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.348140 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl"] Nov 04 01:30:23.400032 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.399976 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7"] Nov 04 01:30:23.402371 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.402343 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00cfe6-29895930-9flk7"] Nov 04 01:30:23.457215 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.457188 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl"] Nov 04 01:30:23.521444 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.521327 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl" Nov 04 01:30:23.572925 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.572888 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0347a89c-4524-4b65-b29a-37a29adfff63-kube-api-access-cp9kj\" (UniqueName: \"kubernetes.io/projected/0347a89c-4524-4b65-b29a-37a29adfff63-kube-api-access-cp9kj\") pod \"0347a89c-4524-4b65-b29a-37a29adfff63\" (UID: \"0347a89c-4524-4b65-b29a-37a29adfff63\") " Nov 04 01:30:23.573101 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.573049 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0347a89c-4524-4b65-b29a-37a29adfff63-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0347a89c-4524-4b65-b29a-37a29adfff63-maas-api-service-ca\") pod \"0347a89c-4524-4b65-b29a-37a29adfff63\" (UID: \"0347a89c-4524-4b65-b29a-37a29adfff63\") " Nov 04 01:30:23.573443 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.573401 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0347a89c-4524-4b65-b29a-37a29adfff63-maas-api-service-ca" pod "0347a89c-4524-4b65-b29a-37a29adfff63" (UID: "0347a89c-4524-4b65-b29a-37a29adfff63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:23.575170 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.575142 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0347a89c-4524-4b65-b29a-37a29adfff63-kube-api-access-cp9kj" pod "0347a89c-4524-4b65-b29a-37a29adfff63" (UID: "0347a89c-4524-4b65-b29a-37a29adfff63"). InnerVolumeSpecName "kube-api-access-cp9kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:23.611247 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.611222 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" Nov 04 01:30:23.673895 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.673861 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daa1061e-0782-40db-96ce-0aee23fa54d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa1061e-0782-40db-96ce-0aee23fa54d4-maas-api-service-ca\") pod \"daa1061e-0782-40db-96ce-0aee23fa54d4\" (UID: \"daa1061e-0782-40db-96ce-0aee23fa54d4\") " Nov 04 01:30:23.674067 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.674020 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa1061e-0782-40db-96ce-0aee23fa54d4-kube-api-access-jlztk\" (UniqueName: \"kubernetes.io/projected/daa1061e-0782-40db-96ce-0aee23fa54d4-kube-api-access-jlztk\") pod \"daa1061e-0782-40db-96ce-0aee23fa54d4\" (UID: \"daa1061e-0782-40db-96ce-0aee23fa54d4\") " Nov 04 01:30:23.674276 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.674229 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa1061e-0782-40db-96ce-0aee23fa54d4-maas-api-service-ca" pod "daa1061e-0782-40db-96ce-0aee23fa54d4" (UID: "daa1061e-0782-40db-96ce-0aee23fa54d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:23.674400 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.674385 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0347a89c-4524-4b65-b29a-37a29adfff63-maas-api-service-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:23.674472 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.674402 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa1061e-0782-40db-96ce-0aee23fa54d4-maas-api-service-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:23.674472 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.674435 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp9kj\" (UniqueName: \"kubernetes.io/projected/0347a89c-4524-4b65-b29a-37a29adfff63-kube-api-access-cp9kj\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:23.676141 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.676120 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa1061e-0782-40db-96ce-0aee23fa54d4-kube-api-access-jlztk" pod "daa1061e-0782-40db-96ce-0aee23fa54d4" (UID: "daa1061e-0782-40db-96ce-0aee23fa54d4"). InnerVolumeSpecName "kube-api-access-jlztk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:23.776188 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:23.776098 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlztk\" (UniqueName: \"kubernetes.io/projected/daa1061e-0782-40db-96ce-0aee23fa54d4-kube-api-access-jlztk\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:24.129530 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:24.129447 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b69798c-73c6-4356-a065-e050b196cf14" path="/var/lib/kubelet/pods/8b69798c-73c6-4356-a065-e050b196cf14/volumes" Nov 04 01:30:24.130017 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:24.130001 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e84e7b4-91ce-406d-96d4-1aa1d6d2e597" path="/var/lib/kubelet/pods/8e84e7b4-91ce-406d-96d4-1aa1d6d2e597/volumes" Nov 04 01:30:24.252750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:24.252723 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl" Nov 04 01:30:24.252750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:24.252741 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl" event={"ID":"0347a89c-4524-4b65-b29a-37a29adfff63","Type":"ContainerDied","Data":"053f6094a754bde18821cf2a02a3349ea6c368d0b3b6e7d5474f83a4c38aa837"} Nov 04 01:30:24.253277 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:24.252802 5995 scope.go:122] "RemoveContainer" containerID="97b7928288e9dfd0ca676c2469bf581861cc42255fcd9d52282abb02184b77c0" Nov 04 01:30:24.254093 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:24.254065 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" Nov 04 01:30:24.254178 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:24.254093 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl" event={"ID":"daa1061e-0782-40db-96ce-0aee23fa54d4","Type":"ContainerDied","Data":"7f5f31879375987cfefe98a9f088a57b6bf610264e23886978f3d5a2265ddb23"} Nov 04 01:30:24.262037 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:24.262020 5995 scope.go:122] "RemoveContainer" containerID="09f2d0069f1cc4a7dbf87a61a0aceb0c7395b3555c52049777992bf9ad3555d8" Nov 04 01:30:24.410702 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:24.410621 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl"] Nov 04 01:30:24.414888 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:24.414864 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1baf54-29895930-4xkwl"] Nov 04 01:30:24.516681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:24.516649 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl"] Nov 04 01:30:24.521007 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:24.520981 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f7183-29895930-7lbgl"] Nov 04 01:30:26.128775 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:26.128742 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0347a89c-4524-4b65-b29a-37a29adfff63" path="/var/lib/kubelet/pods/0347a89c-4524-4b65-b29a-37a29adfff63/volumes" Nov 04 01:30:26.129189 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:26.129176 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daa1061e-0782-40db-96ce-0aee23fa54d4" path="/var/lib/kubelet/pods/daa1061e-0782-40db-96ce-0aee23fa54d4/volumes" Nov 04 01:30:33.591552 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:33.591475 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:30:49.642259 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:30:49.642217 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:30:50.245553 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:50.245517 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f_5600ddae-56c7-4c93-afff-2900207b0819/extract/0.log" Nov 04 01:30:50.247831 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:50.247801 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f_5600ddae-56c7-4c93-afff-2900207b0819/extract/0.log" Nov 04 01:30:50.256583 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:50.256561 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:30:50.258064 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:50.258034 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:30:55.528033 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.527994 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw"] Nov 04 01:30:55.528472 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528458 5995 cpu_manager.go:414] "containerMap: removing container" podUID="8e84e7b4-91ce-406d-96d4-1aa1d6d2e597" containerName="cleanup" Nov 04 01:30:55.528529 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528475 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e84e7b4-91ce-406d-96d4-1aa1d6d2e597" containerName="cleanup" Nov 04 01:30:55.528529 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528489 5995 cpu_manager.go:414] "containerMap: removing container" podUID="0347a89c-4524-4b65-b29a-37a29adfff63" containerName="cleanup" Nov 04 01:30:55.528529 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528494 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0347a89c-4524-4b65-b29a-37a29adfff63" containerName="cleanup" Nov 04 01:30:55.528529 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528509 5995 cpu_manager.go:414] "containerMap: removing container" podUID="8b69798c-73c6-4356-a065-e050b196cf14" containerName="cleanup" Nov 04 01:30:55.528529 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528514 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b69798c-73c6-4356-a065-e050b196cf14" containerName="cleanup" Nov 04 01:30:55.528529 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528521 5995 cpu_manager.go:414] "containerMap: removing container" podUID="66684988-7433-4da1-94fb-8110cf7c31c7" containerName="istio-proxy" Nov 04 01:30:55.528529 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528525 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66684988-7433-4da1-94fb-8110cf7c31c7" containerName="istio-proxy" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528534 5995 cpu_manager.go:414] "containerMap: removing container" podUID="daa1061e-0782-40db-96ce-0aee23fa54d4" containerName="cleanup" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528538 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa1061e-0782-40db-96ce-0aee23fa54d4" containerName="cleanup" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528564 5995 cpu_manager.go:414] "containerMap: removing container" podUID="0347a89c-4524-4b65-b29a-37a29adfff63" containerName="cleanup" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528570 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0347a89c-4524-4b65-b29a-37a29adfff63" containerName="cleanup" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528578 5995 cpu_manager.go:414] "containerMap: removing container" podUID="8e84e7b4-91ce-406d-96d4-1aa1d6d2e597" containerName="cleanup" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528583 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e84e7b4-91ce-406d-96d4-1aa1d6d2e597" containerName="cleanup" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528590 5995 cpu_manager.go:414] "containerMap: removing container" podUID="8b69798c-73c6-4356-a065-e050b196cf14" containerName="cleanup" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528594 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b69798c-73c6-4356-a065-e050b196cf14" containerName="cleanup" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528601 5995 cpu_manager.go:414] "containerMap: removing container" podUID="daa1061e-0782-40db-96ce-0aee23fa54d4" containerName="cleanup" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528605 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa1061e-0782-40db-96ce-0aee23fa54d4" containerName="cleanup" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528675 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa1061e-0782-40db-96ce-0aee23fa54d4" containerName="cleanup" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528682 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="0347a89c-4524-4b65-b29a-37a29adfff63" containerName="cleanup" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528688 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b69798c-73c6-4356-a065-e050b196cf14" containerName="cleanup" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528693 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e84e7b4-91ce-406d-96d4-1aa1d6d2e597" containerName="cleanup" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528699 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa1061e-0782-40db-96ce-0aee23fa54d4" containerName="cleanup" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528708 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e84e7b4-91ce-406d-96d4-1aa1d6d2e597" containerName="cleanup" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528715 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="0347a89c-4524-4b65-b29a-37a29adfff63" containerName="cleanup" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528721 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b69798c-73c6-4356-a065-e050b196cf14" containerName="cleanup" Nov 04 01:30:55.528728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.528726 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="66684988-7433-4da1-94fb-8110cf7c31c7" containerName="istio-proxy" Nov 04 01:30:55.533953 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.533933 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.583198 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.583167 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-00cfe6-openshift-default-dockercfg-5jwvp\"" Nov 04 01:30:55.597132 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.597105 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-envoy\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.597252 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.597141 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-token\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.597252 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.597164 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk2fg\" (UniqueName: \"kubernetes.io/projected/aca25313-a6c6-4930-920f-488a72e6a3f6-kube-api-access-tk2fg\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.597324 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.597283 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-podinfo\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.597357 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.597337 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-data\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.597392 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.597373 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca25313-a6c6-4930-920f-488a72e6a3f6-istiod-ca-cert\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.597471 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.597458 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-workload-socket\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.597512 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.597491 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-credential-socket\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.597575 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.597561 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-workload-certs\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.632687 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.632658 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw"] Nov 04 01:30:55.698999 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.698965 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-data\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.698999 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.698998 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca25313-a6c6-4930-920f-488a72e6a3f6-istiod-ca-cert\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.699243 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.699026 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-workload-socket\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.699243 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.699055 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-credential-socket\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.699243 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.699076 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-workload-certs\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.699243 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.699098 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-envoy\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.699243 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.699126 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-token\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.699243 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.699161 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk2fg\" (UniqueName: \"kubernetes.io/projected/aca25313-a6c6-4930-920f-488a72e6a3f6-kube-api-access-tk2fg\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.699243 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.699233 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-podinfo\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.699628 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.699526 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-credential-socket\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.699628 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.699604 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-data\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.699727 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.699667 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-workload-certs\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.699778 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.699755 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-workload-socket\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.699927 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.699904 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca25313-a6c6-4930-920f-488a72e6a3f6-istiod-ca-cert\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.701538 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.701518 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-envoy\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.701795 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.701779 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-podinfo\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.757239 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.757209 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-token\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.757452 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.757408 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk2fg\" (UniqueName: \"kubernetes.io/projected/aca25313-a6c6-4930-920f-488a72e6a3f6-kube-api-access-tk2fg\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:55.845783 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:55.845705 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:56.268405 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:56.268362 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw"] Nov 04 01:30:56.274217 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:30:56.274184 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca25313_a6c6_4930_920f_488a72e6a3f6.slice/crio-c9be9f2c798d984569aa248064c25bc1fb646657ab47e4c4332be381a07ac426 WatchSource:0}: Error finding container c9be9f2c798d984569aa248064c25bc1fb646657ab47e4c4332be381a07ac426: Status 404 returned error can't find the container with id c9be9f2c798d984569aa248064c25bc1fb646657ab47e4c4332be381a07ac426 Nov 04 01:30:56.276984 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:56.276946 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:30:56.277114 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:56.277029 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:30:56.277114 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:56.277076 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:30:56.444134 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:56.444102 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" event={"ID":"aca25313-a6c6-4930-920f-488a72e6a3f6","Type":"ContainerStarted","Data":"87d3b87013c0a55c8bbcaf7907e8c30692cd11bdccc3032b1fecf6c261fb4a9d"} Nov 04 01:30:56.444285 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:56.444160 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" event={"ID":"aca25313-a6c6-4930-920f-488a72e6a3f6","Type":"ContainerStarted","Data":"c9be9f2c798d984569aa248064c25bc1fb646657ab47e4c4332be381a07ac426"} Nov 04 01:30:56.559663 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:56.559550 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" podStartSLOduration=1.5595335129999999 podStartE2EDuration="1.559533513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:56.557223747 +0000 UTC m=+1206.588957140" watchObservedRunningTime="2026-11-04 01:30:56.559533513 +0000 UTC m=+1206.591266902" Nov 04 01:30:56.846723 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:56.846646 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:30:57.847804 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:57.847769 5995 patch_prober.go:29] interesting pod/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:30:57.848188 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:57.847821 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" podUID="aca25313-a6c6-4930-920f-488a72e6a3f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 04 01:30:58.832474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.832444 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7633-346153"] Nov 04 01:30:58.832868 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.832856 5995 cpu_manager.go:414] "containerMap: removing container" podUID="8e84e7b4-91ce-406d-96d4-1aa1d6d2e597" containerName="cleanup" Nov 04 01:30:58.832910 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.832869 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e84e7b4-91ce-406d-96d4-1aa1d6d2e597" containerName="cleanup" Nov 04 01:30:58.832910 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.832880 5995 cpu_manager.go:414] "containerMap: removing container" podUID="0347a89c-4524-4b65-b29a-37a29adfff63" containerName="cleanup" Nov 04 01:30:58.832910 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.832885 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0347a89c-4524-4b65-b29a-37a29adfff63" containerName="cleanup" Nov 04 01:30:58.832910 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.832900 5995 cpu_manager.go:414] "containerMap: removing container" podUID="8b69798c-73c6-4356-a065-e050b196cf14" containerName="cleanup" Nov 04 01:30:58.832910 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.832906 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b69798c-73c6-4356-a065-e050b196cf14" containerName="cleanup" Nov 04 01:30:58.833058 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.832915 5995 cpu_manager.go:414] "containerMap: removing container" podUID="daa1061e-0782-40db-96ce-0aee23fa54d4" containerName="cleanup" Nov 04 01:30:58.833058 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.832919 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa1061e-0782-40db-96ce-0aee23fa54d4" containerName="cleanup" Nov 04 01:30:58.833058 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.833038 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa1061e-0782-40db-96ce-0aee23fa54d4" containerName="cleanup" Nov 04 01:30:58.833058 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.833046 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e84e7b4-91ce-406d-96d4-1aa1d6d2e597" containerName="cleanup" Nov 04 01:30:58.833058 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.833053 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="0347a89c-4524-4b65-b29a-37a29adfff63" containerName="cleanup" Nov 04 01:30:58.833202 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.833064 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b69798c-73c6-4356-a065-e050b196cf14" containerName="cleanup" Nov 04 01:30:58.835726 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.835707 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-346153" Nov 04 01:30:58.847534 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.847506 5995 patch_prober.go:29] interesting pod/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:30:58.847701 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.847680 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" podUID="aca25313-a6c6-4930-920f-488a72e6a3f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 04 01:30:58.889881 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.889850 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7633-346153"] Nov 04 01:30:58.931201 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.931170 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vcvq\" (UniqueName: \"kubernetes.io/projected/9d371f5a-b7b1-454c-865c-75ea65421758-kube-api-access-6vcvq\") pod \"test-curl-7633-346153\" (UID: \"9d371f5a-b7b1-454c-865c-75ea65421758\") " pod="opendatahub/test-curl-7633-346153" Nov 04 01:30:58.933643 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.933621 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 04 01:30:58.933742 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.933656 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 04 01:30:58.933742 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:58.933661 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8vc6z\"" Nov 04 01:30:59.032752 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:59.032718 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vcvq\" (UniqueName: \"kubernetes.io/projected/9d371f5a-b7b1-454c-865c-75ea65421758-kube-api-access-6vcvq\") pod \"test-curl-7633-346153\" (UID: \"9d371f5a-b7b1-454c-865c-75ea65421758\") " pod="opendatahub/test-curl-7633-346153" Nov 04 01:30:59.090572 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:59.090493 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vcvq\" (UniqueName: \"kubernetes.io/projected/9d371f5a-b7b1-454c-865c-75ea65421758-kube-api-access-6vcvq\") pod \"test-curl-7633-346153\" (UID: \"9d371f5a-b7b1-454c-865c-75ea65421758\") " pod="opendatahub/test-curl-7633-346153" Nov 04 01:30:59.147758 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:59.147730 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-346153" Nov 04 01:30:59.540544 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:59.540512 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7633-346153"] Nov 04 01:30:59.547086 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:30:59.547058 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d371f5a_b7b1_454c_865c_75ea65421758.slice/crio-2acea6fc5c1daa4e16793d961b5a6784fb57d85595569a2eca1e9ac5099b0524 WatchSource:0}: Error finding container 2acea6fc5c1daa4e16793d961b5a6784fb57d85595569a2eca1e9ac5099b0524: Status 404 returned error can't find the container with id 2acea6fc5c1daa4e16793d961b5a6784fb57d85595569a2eca1e9ac5099b0524 Nov 04 01:30:59.848491 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:59.846636 5995 patch_prober.go:29] interesting pod/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:30:59.848491 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:30:59.846732 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" podUID="aca25313-a6c6-4930-920f-488a72e6a3f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 04 01:31:00.491121 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:00.491085 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7633-346153" event={"ID":"9d371f5a-b7b1-454c-865c-75ea65421758","Type":"ContainerStarted","Data":"2acea6fc5c1daa4e16793d961b5a6784fb57d85595569a2eca1e9ac5099b0524"} Nov 04 01:31:00.846770 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:00.846687 5995 patch_prober.go:29] interesting pod/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:31:00.846770 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:00.846740 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" podUID="aca25313-a6c6-4930-920f-488a72e6a3f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 04 01:31:01.847086 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:01.847050 5995 patch_prober.go:29] interesting pod/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:31:01.847539 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:01.847112 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" podUID="aca25313-a6c6-4930-920f-488a72e6a3f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 04 01:31:02.846681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:02.846643 5995 patch_prober.go:29] interesting pod/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:31:02.846853 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:02.846704 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" podUID="aca25313-a6c6-4930-920f-488a72e6a3f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 04 01:31:03.513559 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:03.513518 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7633-346153" event={"ID":"9d371f5a-b7b1-454c-865c-75ea65421758","Type":"ContainerStarted","Data":"2b2c6521efb49f5f970477bad88e262493e9dcd1af3eed5788528a49e03e3adf"} Nov 04 01:31:03.622098 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:03.621039 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7633-346153" podStartSLOduration=2.307381477 podStartE2EDuration="5.621020624s" totalImagesPullingTime="3.313639147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:03.619881418 +0000 UTC m=+1213.651614807" watchObservedRunningTime="2026-11-04 01:31:03.621020624 +0000 UTC m=+1213.652754007" Nov 04 01:31:03.850595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:03.850521 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:31:03.850808 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:03.850787 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:31:03.851522 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:03.851498 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:31:05.123301 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:05.123268 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7633-346153"] Nov 04 01:31:05.126315 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:05.126283 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7633-346153"] Nov 04 01:31:05.288940 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:05.288904 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7633-ac7032"] Nov 04 01:31:05.289348 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:05.289335 5995 cpu_manager.go:414] "containerMap: removing container" podUID="9d371f5a-b7b1-454c-865c-75ea65421758" containerName="test-curl-7633-346153" Nov 04 01:31:05.289389 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:05.289349 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d371f5a-b7b1-454c-865c-75ea65421758" containerName="test-curl-7633-346153" Nov 04 01:31:05.289490 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:05.289480 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d371f5a-b7b1-454c-865c-75ea65421758" containerName="test-curl-7633-346153" Nov 04 01:31:05.291867 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:05.291847 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-ac7032" Nov 04 01:31:05.346684 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:05.346651 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7633-ac7032"] Nov 04 01:31:05.429948 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:05.429866 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwg6x\" (UniqueName: \"kubernetes.io/projected/4a58aae6-f0a1-4c30-8845-23b1d1574c0e-kube-api-access-jwg6x\") pod \"test-curl-7633-ac7032\" (UID: \"4a58aae6-f0a1-4c30-8845-23b1d1574c0e\") " pod="opendatahub/test-curl-7633-ac7032" Nov 04 01:31:05.525199 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:05.525155 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7633-346153" podUID="9d371f5a-b7b1-454c-865c-75ea65421758" containerName="test-curl-7633-346153" containerID="cri-o://2b2c6521efb49f5f970477bad88e262493e9dcd1af3eed5788528a49e03e3adf" gracePeriod=2 Nov 04 01:31:05.530967 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:05.530947 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwg6x\" (UniqueName: \"kubernetes.io/projected/4a58aae6-f0a1-4c30-8845-23b1d1574c0e-kube-api-access-jwg6x\") pod \"test-curl-7633-ac7032\" (UID: \"4a58aae6-f0a1-4c30-8845-23b1d1574c0e\") " pod="opendatahub/test-curl-7633-ac7032" Nov 04 01:31:05.573566 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:31:05.573535 5995 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7633-346153\" is forbidden: User \"system:node:ip-10-0-138-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-179.us-west-2.compute.internal' and this object" podUID="9d371f5a-b7b1-454c-865c-75ea65421758" pod="opendatahub/test-curl-7633-346153" Nov 04 01:31:05.588630 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:05.588612 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwg6x\" (UniqueName: \"kubernetes.io/projected/4a58aae6-f0a1-4c30-8845-23b1d1574c0e-kube-api-access-jwg6x\") pod \"test-curl-7633-ac7032\" (UID: \"4a58aae6-f0a1-4c30-8845-23b1d1574c0e\") " pod="opendatahub/test-curl-7633-ac7032" Nov 04 01:31:05.602529 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:05.602507 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-ac7032" Nov 04 01:31:05.966874 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:05.966841 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7633-ac7032"] Nov 04 01:31:05.973165 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:31:05.973136 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a58aae6_f0a1_4c30_8845_23b1d1574c0e.slice/crio-2e3b28bf37e002679339456fb636727c0eb1920bf8585f3b7c59b9268ab906f1 WatchSource:0}: Error finding container 2e3b28bf37e002679339456fb636727c0eb1920bf8585f3b7c59b9268ab906f1: Status 404 returned error can't find the container with id 2e3b28bf37e002679339456fb636727c0eb1920bf8585f3b7c59b9268ab906f1 Nov 04 01:31:06.532152 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:06.532112 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7633-ac7032" event={"ID":"4a58aae6-f0a1-4c30-8845-23b1d1574c0e","Type":"ContainerStarted","Data":"ab081b12153fc70a8e9d47707246713cf46edd506660cf002ddf35730cc52b67"} Nov 04 01:31:06.532152 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:06.532158 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7633-ac7032" event={"ID":"4a58aae6-f0a1-4c30-8845-23b1d1574c0e","Type":"ContainerStarted","Data":"2e3b28bf37e002679339456fb636727c0eb1920bf8585f3b7c59b9268ab906f1"} Nov 04 01:31:06.639389 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:06.639331 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7633-ac7032" podStartSLOduration=1.639314577 podStartE2EDuration="1.639314577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:06.638723562 +0000 UTC m=+1216.670456951" watchObservedRunningTime="2026-11-04 01:31:06.639314577 +0000 UTC m=+1216.671047966" Nov 04 01:31:07.726812 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:07.726787 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-346153" Nov 04 01:31:07.853744 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:07.853668 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d371f5a-b7b1-454c-865c-75ea65421758-kube-api-access-6vcvq\" (UniqueName: \"kubernetes.io/projected/9d371f5a-b7b1-454c-865c-75ea65421758-kube-api-access-6vcvq\") pod \"9d371f5a-b7b1-454c-865c-75ea65421758\" (UID: \"9d371f5a-b7b1-454c-865c-75ea65421758\") " Nov 04 01:31:07.855831 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:07.855804 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d371f5a-b7b1-454c-865c-75ea65421758-kube-api-access-6vcvq" pod "9d371f5a-b7b1-454c-865c-75ea65421758" (UID: "9d371f5a-b7b1-454c-865c-75ea65421758"). InnerVolumeSpecName "kube-api-access-6vcvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:07.955432 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:07.955395 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vcvq\" (UniqueName: \"kubernetes.io/projected/9d371f5a-b7b1-454c-865c-75ea65421758-kube-api-access-6vcvq\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:08.128853 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:08.128779 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d371f5a-b7b1-454c-865c-75ea65421758" path="/var/lib/kubelet/pods/9d371f5a-b7b1-454c-865c-75ea65421758/volumes" Nov 04 01:31:08.261491 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:08.261455 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7633-ac7032"] Nov 04 01:31:08.264542 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:08.264512 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7633-ac7032"] Nov 04 01:31:08.543934 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:08.543895 5995 generic.go:356] "Generic (PLEG): container finished" podID="9d371f5a-b7b1-454c-865c-75ea65421758" containerID="2b2c6521efb49f5f970477bad88e262493e9dcd1af3eed5788528a49e03e3adf" exitCode=137 Nov 04 01:31:08.544101 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:08.543952 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-346153" Nov 04 01:31:08.544101 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:08.543995 5995 scope.go:122] "RemoveContainer" containerID="2b2c6521efb49f5f970477bad88e262493e9dcd1af3eed5788528a49e03e3adf" Nov 04 01:31:08.544213 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:08.544111 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7633-ac7032" podUID="4a58aae6-f0a1-4c30-8845-23b1d1574c0e" containerName="test-curl-7633-ac7032" containerID="cri-o://ab081b12153fc70a8e9d47707246713cf46edd506660cf002ddf35730cc52b67" gracePeriod=2 Nov 04 01:31:08.552501 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:08.552484 5995 scope.go:122] "RemoveContainer" containerID="2b2c6521efb49f5f970477bad88e262493e9dcd1af3eed5788528a49e03e3adf" Nov 04 01:31:08.552789 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:31:08.552761 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2c6521efb49f5f970477bad88e262493e9dcd1af3eed5788528a49e03e3adf\": container with ID starting with 2b2c6521efb49f5f970477bad88e262493e9dcd1af3eed5788528a49e03e3adf not found: ID does not exist" containerID="2b2c6521efb49f5f970477bad88e262493e9dcd1af3eed5788528a49e03e3adf" Nov 04 01:31:08.552843 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:08.552793 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2c6521efb49f5f970477bad88e262493e9dcd1af3eed5788528a49e03e3adf"} err="failed to get container status \"2b2c6521efb49f5f970477bad88e262493e9dcd1af3eed5788528a49e03e3adf\": rpc error: code = NotFound desc = could not find container \"2b2c6521efb49f5f970477bad88e262493e9dcd1af3eed5788528a49e03e3adf\": container with ID starting with 2b2c6521efb49f5f970477bad88e262493e9dcd1af3eed5788528a49e03e3adf not found: ID does not exist" Nov 04 01:31:08.592144 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:31:08.592115 5995 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7633-ac7032\" is forbidden: User \"system:node:ip-10-0-138-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-179.us-west-2.compute.internal' and this object" podUID="4a58aae6-f0a1-4c30-8845-23b1d1574c0e" pod="opendatahub/test-curl-7633-ac7032" Nov 04 01:31:08.641998 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:31:08.641965 5995 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7633-ac7032\" is forbidden: User \"system:node:ip-10-0-138-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-179.us-west-2.compute.internal' and this object" podUID="4a58aae6-f0a1-4c30-8845-23b1d1574c0e" pod="opendatahub/test-curl-7633-ac7032" Nov 04 01:31:10.172861 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:31:10.172821 5995 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7633-ac7032\" is forbidden: User \"system:node:ip-10-0-138-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-179.us-west-2.compute.internal' and this object" podUID="4a58aae6-f0a1-4c30-8845-23b1d1574c0e" pod="opendatahub/test-curl-7633-ac7032" Nov 04 01:31:10.757700 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:10.757670 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-ac7032" Nov 04 01:31:10.888465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:10.888371 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a58aae6-f0a1-4c30-8845-23b1d1574c0e-kube-api-access-jwg6x\" (UniqueName: \"kubernetes.io/projected/4a58aae6-f0a1-4c30-8845-23b1d1574c0e-kube-api-access-jwg6x\") pod \"4a58aae6-f0a1-4c30-8845-23b1d1574c0e\" (UID: \"4a58aae6-f0a1-4c30-8845-23b1d1574c0e\") " Nov 04 01:31:10.890391 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:10.890372 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a58aae6-f0a1-4c30-8845-23b1d1574c0e-kube-api-access-jwg6x" pod "4a58aae6-f0a1-4c30-8845-23b1d1574c0e" (UID: "4a58aae6-f0a1-4c30-8845-23b1d1574c0e"). InnerVolumeSpecName "kube-api-access-jwg6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:10.990088 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:10.990056 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwg6x\" (UniqueName: \"kubernetes.io/projected/4a58aae6-f0a1-4c30-8845-23b1d1574c0e-kube-api-access-jwg6x\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:11.562586 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:11.562550 5995 generic.go:356] "Generic (PLEG): container finished" podID="4a58aae6-f0a1-4c30-8845-23b1d1574c0e" containerID="ab081b12153fc70a8e9d47707246713cf46edd506660cf002ddf35730cc52b67" exitCode=137 Nov 04 01:31:11.562992 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:11.562596 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7633-ac7032" Nov 04 01:31:11.562992 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:11.562635 5995 scope.go:122] "RemoveContainer" containerID="ab081b12153fc70a8e9d47707246713cf46edd506660cf002ddf35730cc52b67" Nov 04 01:31:11.571621 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:11.571602 5995 scope.go:122] "RemoveContainer" containerID="ab081b12153fc70a8e9d47707246713cf46edd506660cf002ddf35730cc52b67" Nov 04 01:31:11.571870 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:31:11.571850 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab081b12153fc70a8e9d47707246713cf46edd506660cf002ddf35730cc52b67\": container with ID starting with ab081b12153fc70a8e9d47707246713cf46edd506660cf002ddf35730cc52b67 not found: ID does not exist" containerID="ab081b12153fc70a8e9d47707246713cf46edd506660cf002ddf35730cc52b67" Nov 04 01:31:11.571909 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:11.571880 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab081b12153fc70a8e9d47707246713cf46edd506660cf002ddf35730cc52b67"} err="failed to get container status \"ab081b12153fc70a8e9d47707246713cf46edd506660cf002ddf35730cc52b67\": rpc error: code = NotFound desc = could not find container \"ab081b12153fc70a8e9d47707246713cf46edd506660cf002ddf35730cc52b67\": container with ID starting with ab081b12153fc70a8e9d47707246713cf46edd506660cf002ddf35730cc52b67 not found: ID does not exist" Nov 04 01:31:11.618527 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:31:11.618492 5995 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7633-ac7032\" is forbidden: User \"system:node:ip-10-0-138-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-179.us-west-2.compute.internal' and this object" podUID="4a58aae6-f0a1-4c30-8845-23b1d1574c0e" pod="opendatahub/test-curl-7633-ac7032" Nov 04 01:31:11.982183 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:11.982149 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn"] Nov 04 01:31:11.982638 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:11.982625 5995 cpu_manager.go:414] "containerMap: removing container" podUID="4a58aae6-f0a1-4c30-8845-23b1d1574c0e" containerName="test-curl-7633-ac7032" Nov 04 01:31:11.982692 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:11.982639 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a58aae6-f0a1-4c30-8845-23b1d1574c0e" containerName="test-curl-7633-ac7032" Nov 04 01:31:11.982741 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:11.982733 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a58aae6-f0a1-4c30-8845-23b1d1574c0e" containerName="test-curl-7633-ac7032" Nov 04 01:31:11.985657 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:11.985637 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.033748 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:31:12.033713 5995 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7633-ac7032\" is forbidden: User \"system:node:ip-10-0-138-179.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-179.us-west-2.compute.internal' and this object" podUID="4a58aae6-f0a1-4c30-8845-23b1d1574c0e" pod="opendatahub/test-curl-7633-ac7032" Nov 04 01:31:12.034937 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.034915 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1baf54\"/\"openshift-service-ca.crt\"" Nov 04 01:31:12.035052 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.035004 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1baf54\"/\"kube-root-ca.crt\"" Nov 04 01:31:12.080789 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.080765 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1baf54\"/\"distinct-w1-1baf54-kserve-self-signed-certs\"" Nov 04 01:31:12.081705 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.081681 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1baf54\"/\"default-dockercfg-4g9z9\"" Nov 04 01:31:12.084677 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.084654 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn"] Nov 04 01:31:12.101738 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.101709 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-home\") pod \"distinct-w1-1baf54-kserve-6f746d7785-rflcn\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.101851 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.101822 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-model-cache\") pod \"distinct-w1-1baf54-kserve-6f746d7785-rflcn\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.101889 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.101867 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnv8x\" (UniqueName: \"kubernetes.io/projected/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-kube-api-access-pnv8x\") pod \"distinct-w1-1baf54-kserve-6f746d7785-rflcn\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.101933 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.101923 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-tls-certs\") pod \"distinct-w1-1baf54-kserve-6f746d7785-rflcn\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.102013 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.101997 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-dshm\") pod \"distinct-w1-1baf54-kserve-6f746d7785-rflcn\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.102051 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.102029 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-tmp-dir\") pod \"distinct-w1-1baf54-kserve-6f746d7785-rflcn\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.130977 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.130944 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a58aae6-f0a1-4c30-8845-23b1d1574c0e" path="/var/lib/kubelet/pods/4a58aae6-f0a1-4c30-8845-23b1d1574c0e/volumes" Nov 04 01:31:12.202803 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.202772 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-model-cache\") pod \"distinct-w1-1baf54-kserve-6f746d7785-rflcn\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.202803 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.202807 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnv8x\" (UniqueName: \"kubernetes.io/projected/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-kube-api-access-pnv8x\") pod \"distinct-w1-1baf54-kserve-6f746d7785-rflcn\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.203020 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.202929 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-tls-certs\") pod \"distinct-w1-1baf54-kserve-6f746d7785-rflcn\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.203020 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.202979 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-dshm\") pod \"distinct-w1-1baf54-kserve-6f746d7785-rflcn\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.203095 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.203019 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-tmp-dir\") pod \"distinct-w1-1baf54-kserve-6f746d7785-rflcn\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.203095 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.203042 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-home\") pod \"distinct-w1-1baf54-kserve-6f746d7785-rflcn\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.203189 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.203170 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-model-cache\") pod \"distinct-w1-1baf54-kserve-6f746d7785-rflcn\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.203363 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.203350 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-tmp-dir\") pod \"distinct-w1-1baf54-kserve-6f746d7785-rflcn\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.203457 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.203434 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-home\") pod \"distinct-w1-1baf54-kserve-6f746d7785-rflcn\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.205154 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.205137 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-dshm\") pod \"distinct-w1-1baf54-kserve-6f746d7785-rflcn\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.205512 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.205494 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-tls-certs\") pod \"distinct-w1-1baf54-kserve-6f746d7785-rflcn\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.260546 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.260481 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnv8x\" (UniqueName: \"kubernetes.io/projected/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-kube-api-access-pnv8x\") pod \"distinct-w1-1baf54-kserve-6f746d7785-rflcn\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.296669 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.296641 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:12.670826 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:12.670743 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn"] Nov 04 01:31:12.676255 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:31:12.676225 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bbe9a41_a5fc_4ee3_98e2_18be1232cffb.slice/crio-4892fedc615e4057fb4e1de0ab085efc34dcf7681d97bfd92fc8acdfdbe6338d WatchSource:0}: Error finding container 4892fedc615e4057fb4e1de0ab085efc34dcf7681d97bfd92fc8acdfdbe6338d: Status 404 returned error can't find the container with id 4892fedc615e4057fb4e1de0ab085efc34dcf7681d97bfd92fc8acdfdbe6338d Nov 04 01:31:13.582737 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:13.582696 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" event={"ID":"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb","Type":"ContainerStarted","Data":"68d49da2a401828d49678a3b1cd7a2e10dbbbbc9319c567b6776033c7ea10aaa"} Nov 04 01:31:13.582944 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:13.582744 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" event={"ID":"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb","Type":"ContainerStarted","Data":"4892fedc615e4057fb4e1de0ab085efc34dcf7681d97bfd92fc8acdfdbe6338d"} Nov 04 01:31:13.695756 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:13.695707 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" podStartSLOduration=2.695692949 podStartE2EDuration="2.695692949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:13.694591097 +0000 UTC m=+1223.726324486" watchObservedRunningTime="2026-11-04 01:31:13.695692949 +0000 UTC m=+1223.727426338" Nov 04 01:31:14.297756 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:14.297721 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:19.643734 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:31:19.643699 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:31:19.856022 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:19.855986 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:31:20.728903 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.728871 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw"] Nov 04 01:31:20.731904 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.731883 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.783597 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.783563 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-5520b3-openshift-default-dockercfg-gvh7s\"" Nov 04 01:31:20.784040 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.784018 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-workload-certs\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.784116 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.784054 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-credential-socket\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.784175 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.784129 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-istio-envoy\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.784175 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.784167 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2jpq\" (UniqueName: \"kubernetes.io/projected/8fa987f7-090b-477e-9779-008bcc81642c-kube-api-access-g2jpq\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.784278 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.784195 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa987f7-090b-477e-9779-008bcc81642c-istio-podinfo\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.784278 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.784245 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa987f7-090b-477e-9779-008bcc81642c-istio-token\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.784278 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.784269 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-workload-socket\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.784509 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.784289 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa987f7-090b-477e-9779-008bcc81642c-istiod-ca-cert\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.784509 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.784316 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-istio-data\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.838239 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.838197 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw"] Nov 04 01:31:20.888707 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.888643 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2jpq\" (UniqueName: \"kubernetes.io/projected/8fa987f7-090b-477e-9779-008bcc81642c-kube-api-access-g2jpq\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.889159 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.889116 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa987f7-090b-477e-9779-008bcc81642c-istio-podinfo\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.889276 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.889203 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa987f7-090b-477e-9779-008bcc81642c-istio-token\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.889276 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.889262 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-workload-socket\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.889384 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.889290 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa987f7-090b-477e-9779-008bcc81642c-istiod-ca-cert\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.889384 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.889322 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-istio-data\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.889526 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.889392 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-workload-certs\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.889526 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.889436 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-credential-socket\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.890900 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.890874 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-istio-envoy\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.895612 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.895558 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-credential-socket\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.895890 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.895852 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-workload-socket\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.897565 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.897537 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa987f7-090b-477e-9779-008bcc81642c-istiod-ca-cert\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.898079 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.898056 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-istio-envoy\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.899851 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.899823 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-istio-data\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.899971 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.899955 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-workload-certs\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.900645 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.900624 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa987f7-090b-477e-9779-008bcc81642c-istio-podinfo\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.949067 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.949035 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2jpq\" (UniqueName: \"kubernetes.io/projected/8fa987f7-090b-477e-9779-008bcc81642c-kube-api-access-g2jpq\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:20.950783 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:20.950758 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa987f7-090b-477e-9779-008bcc81642c-istio-token\") pod \"e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:21.044978 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:21.044640 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:21.472366 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:21.472332 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw"] Nov 04 01:31:21.478922 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:31:21.478892 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa987f7_090b_477e_9779_008bcc81642c.slice/crio-7454314aa9247fabd1e19599e94f707a4479e0239a25a804b103a463bcdbc646 WatchSource:0}: Error finding container 7454314aa9247fabd1e19599e94f707a4479e0239a25a804b103a463bcdbc646: Status 404 returned error can't find the container with id 7454314aa9247fabd1e19599e94f707a4479e0239a25a804b103a463bcdbc646 Nov 04 01:31:21.481335 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:21.481301 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:31:21.481453 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:21.481369 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:31:21.481453 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:21.481399 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:31:21.635429 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:21.635386 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" event={"ID":"8fa987f7-090b-477e-9779-008bcc81642c","Type":"ContainerStarted","Data":"71c6227ee4977db1e94a7f1c18b2b2abf859a564cfc7dbe35ef71e8152562f55"} Nov 04 01:31:21.635618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:21.635459 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" event={"ID":"8fa987f7-090b-477e-9779-008bcc81642c","Type":"ContainerStarted","Data":"7454314aa9247fabd1e19599e94f707a4479e0239a25a804b103a463bcdbc646"} Nov 04 01:31:21.751853 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:21.751749 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" podStartSLOduration=1.751735257 podStartE2EDuration="1.751735257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:21.750515276 +0000 UTC m=+1231.782248665" watchObservedRunningTime="2026-11-04 01:31:21.751735257 +0000 UTC m=+1231.783468646" Nov 04 01:31:22.046410 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:22.046321 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:22.051524 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:22.051500 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:22.297383 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:22.297300 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:22.309784 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:22.309754 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:22.641673 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:22.641593 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:22.642316 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:22.642296 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:31:22.652245 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:22.652217 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:31:28.428970 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.428932 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt"] Nov 04 01:31:28.432684 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.432663 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt" Nov 04 01:31:28.485142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.485109 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48618274-e82a-4031-aff6-f2ff37064488-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt\" (UID: \"48618274-e82a-4031-aff6-f2ff37064488\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt" Nov 04 01:31:28.485320 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.485275 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhq2l\" (UniqueName: \"kubernetes.io/projected/48618274-e82a-4031-aff6-f2ff37064488-kube-api-access-mhq2l\") pod \"payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt\" (UID: \"48618274-e82a-4031-aff6-f2ff37064488\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt" Nov 04 01:31:28.486605 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.486574 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-5520b3\"" Nov 04 01:31:28.528348 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.528310 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-5520b3-dockercfg-rb8hm\"" Nov 04 01:31:28.530612 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.530591 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt"] Nov 04 01:31:28.608099 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.608057 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhq2l\" (UniqueName: \"kubernetes.io/projected/48618274-e82a-4031-aff6-f2ff37064488-kube-api-access-mhq2l\") pod \"payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt\" (UID: \"48618274-e82a-4031-aff6-f2ff37064488\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt" Nov 04 01:31:28.608460 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.608433 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48618274-e82a-4031-aff6-f2ff37064488-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt\" (UID: \"48618274-e82a-4031-aff6-f2ff37064488\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt" Nov 04 01:31:28.609584 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.609561 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48618274-e82a-4031-aff6-f2ff37064488-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt\" (UID: \"48618274-e82a-4031-aff6-f2ff37064488\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt" Nov 04 01:31:28.621738 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.619122 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn"] Nov 04 01:31:28.623233 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.623208 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn" Nov 04 01:31:28.656365 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.656331 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn"] Nov 04 01:31:28.667688 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.667655 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhq2l\" (UniqueName: \"kubernetes.io/projected/48618274-e82a-4031-aff6-f2ff37064488-kube-api-access-mhq2l\") pod \"payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt\" (UID: \"48618274-e82a-4031-aff6-f2ff37064488\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt" Nov 04 01:31:28.709397 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.709366 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8gfp\" (UniqueName: \"kubernetes.io/projected/0957c760-9805-417a-ab0f-1289fb7a70ad-kube-api-access-d8gfp\") pod \"payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn\" (UID: \"0957c760-9805-417a-ab0f-1289fb7a70ad\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn" Nov 04 01:31:28.709545 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.709494 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0957c760-9805-417a-ab0f-1289fb7a70ad-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn\" (UID: \"0957c760-9805-417a-ab0f-1289fb7a70ad\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn" Nov 04 01:31:28.745601 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.745572 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt" Nov 04 01:31:28.811866 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.810436 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8gfp\" (UniqueName: \"kubernetes.io/projected/0957c760-9805-417a-ab0f-1289fb7a70ad-kube-api-access-d8gfp\") pod \"payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn\" (UID: \"0957c760-9805-417a-ab0f-1289fb7a70ad\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn" Nov 04 01:31:28.811866 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.810550 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0957c760-9805-417a-ab0f-1289fb7a70ad-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn\" (UID: \"0957c760-9805-417a-ab0f-1289fb7a70ad\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn" Nov 04 01:31:28.811866 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.811520 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0957c760-9805-417a-ab0f-1289fb7a70ad-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn\" (UID: \"0957c760-9805-417a-ab0f-1289fb7a70ad\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn" Nov 04 01:31:28.869675 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.869644 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8gfp\" (UniqueName: \"kubernetes.io/projected/0957c760-9805-417a-ab0f-1289fb7a70ad-kube-api-access-d8gfp\") pod \"payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn\" (UID: \"0957c760-9805-417a-ab0f-1289fb7a70ad\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn" Nov 04 01:31:28.962351 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:28.962265 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn" Nov 04 01:31:29.116885 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:29.116854 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt"] Nov 04 01:31:29.123031 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:31:29.122993 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48618274_e82a_4031_aff6_f2ff37064488.slice/crio-578f7896fd2a6ed7dfc54f00256b8d4907b9b714ba21fe5412fec71a07b29dc4 WatchSource:0}: Error finding container 578f7896fd2a6ed7dfc54f00256b8d4907b9b714ba21fe5412fec71a07b29dc4: Status 404 returned error can't find the container with id 578f7896fd2a6ed7dfc54f00256b8d4907b9b714ba21fe5412fec71a07b29dc4 Nov 04 01:31:29.335287 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:29.335201 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn"] Nov 04 01:31:29.340451 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:31:29.340407 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0957c760_9805_417a_ab0f_1289fb7a70ad.slice/crio-2d782afaf70515011de59ddb1c6581d56c101113f7ef1922cd940180b00cb6f5 WatchSource:0}: Error finding container 2d782afaf70515011de59ddb1c6581d56c101113f7ef1922cd940180b00cb6f5: Status 404 returned error can't find the container with id 2d782afaf70515011de59ddb1c6581d56c101113f7ef1922cd940180b00cb6f5 Nov 04 01:31:29.692746 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:29.692719 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt" event={"ID":"48618274-e82a-4031-aff6-f2ff37064488","Type":"ContainerStarted","Data":"8b53f40eb5eddc2099492dd7096e103aad91c8e1038e7a01d614451b4dfe034b"} Nov 04 01:31:29.693164 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:29.692768 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt" event={"ID":"48618274-e82a-4031-aff6-f2ff37064488","Type":"ContainerStarted","Data":"578f7896fd2a6ed7dfc54f00256b8d4907b9b714ba21fe5412fec71a07b29dc4"} Nov 04 01:31:29.693164 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:29.692905 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt" Nov 04 01:31:29.694674 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:29.694644 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn" event={"ID":"0957c760-9805-417a-ab0f-1289fb7a70ad","Type":"ContainerStarted","Data":"9a1d67788156e949b6cc16bd2d6e7e4db0f7ac864313ed233cf0541bbde73562"} Nov 04 01:31:29.694809 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:29.694694 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn" event={"ID":"0957c760-9805-417a-ab0f-1289fb7a70ad","Type":"ContainerStarted","Data":"2d782afaf70515011de59ddb1c6581d56c101113f7ef1922cd940180b00cb6f5"} Nov 04 01:31:29.694809 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:29.694722 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn" Nov 04 01:31:29.804162 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:29.804111 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt" podStartSLOduration=1.8040968309999998 podStartE2EDuration="1.804096831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:29.801612432 +0000 UTC m=+1239.833345822" watchObservedRunningTime="2026-11-04 01:31:29.804096831 +0000 UTC m=+1239.835830219" Nov 04 01:31:29.912292 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:29.912238 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn" podStartSLOduration=1.9122261040000001 podStartE2EDuration="1.912226104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:29.91017111 +0000 UTC m=+1239.941904499" watchObservedRunningTime="2026-11-04 01:31:29.912226104 +0000 UTC m=+1239.943959517" Nov 04 01:31:40.704216 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:40.704174 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt" Nov 04 01:31:40.704719 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:40.704261 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn" Nov 04 01:31:41.387962 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:31:41.387926 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:31:49.641263 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:31:49.641217 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:32:14.356502 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:14.356423 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:32:19.643809 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:32:19.643762 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:32:23.683555 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:23.683521 5995 patch_prober.go:29] interesting pod/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:32:23.683954 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:23.683581 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" podUID="8fa987f7-090b-477e-9779-008bcc81642c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:25.036756 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:25.036723 5995 generic.go:356] "Generic (PLEG): container finished" podID="8fa987f7-090b-477e-9779-008bcc81642c" containerID="71c6227ee4977db1e94a7f1c18b2b2abf859a564cfc7dbe35ef71e8152562f55" exitCode=0 Nov 04 01:32:25.037176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:25.036798 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" event={"ID":"8fa987f7-090b-477e-9779-008bcc81642c","Type":"ContainerDied","Data":"71c6227ee4977db1e94a7f1c18b2b2abf859a564cfc7dbe35ef71e8152562f55"} Nov 04 01:32:25.037220 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:25.037192 5995 scope.go:122] "RemoveContainer" containerID="71c6227ee4977db1e94a7f1c18b2b2abf859a564cfc7dbe35ef71e8152562f55" Nov 04 01:32:25.038509 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:25.038481 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:32:25.038573 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:25.038545 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:32:25.038616 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:25.038577 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:32:26.060041 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:26.059994 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" event={"ID":"8fa987f7-090b-477e-9779-008bcc81642c","Type":"ContainerStarted","Data":"5abe425217937dd6d60a7af1fa9fc427474ddfa5ae9df3865bff71b465337f63"} Nov 04 01:32:27.047609 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:27.046682 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:32:27.047609 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:27.046811 5995 patch_prober.go:29] interesting pod/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 04 01:32:27.047609 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:27.046856 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" podUID="8fa987f7-090b-477e-9779-008bcc81642c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 04 01:32:28.046038 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:28.046001 5995 patch_prober.go:29] interesting pod/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 04 01:32:28.046505 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:28.046094 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" podUID="8fa987f7-090b-477e-9779-008bcc81642c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 04 01:32:29.045771 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:29.045735 5995 patch_prober.go:29] interesting pod/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 04 01:32:29.045995 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:29.045825 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" podUID="8fa987f7-090b-477e-9779-008bcc81642c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 04 01:32:30.046521 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:30.046480 5995 patch_prober.go:29] interesting pod/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 04 01:32:30.046912 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:30.046535 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" podUID="8fa987f7-090b-477e-9779-008bcc81642c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 04 01:32:31.054780 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:31.054749 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:32:31.055723 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:31.055696 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:32:31.056762 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:31.056741 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:32:41.189325 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.189282 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd"] Nov 04 01:32:41.192540 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.192515 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.247143 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.247107 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-credential-socket\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.247143 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.247141 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c7bbe4-d255-480e-a031-b9987c89f47b-istiod-ca-cert\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.247378 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.247172 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-data\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.247378 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.247246 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-podinfo\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.247378 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.247357 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-envoy\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.247512 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.247461 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-workload-socket\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.247560 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.247514 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g6rg\" (UniqueName: \"kubernetes.io/projected/94c7bbe4-d255-480e-a031-b9987c89f47b-kube-api-access-8g6rg\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.247560 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.247542 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-workload-certs\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.247624 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.247570 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-token\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.292733 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.292694 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd"] Nov 04 01:32:41.348532 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.348491 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-envoy\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.348713 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.348580 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-workload-socket\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.348713 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.348627 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g6rg\" (UniqueName: \"kubernetes.io/projected/94c7bbe4-d255-480e-a031-b9987c89f47b-kube-api-access-8g6rg\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.348713 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.348651 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-workload-certs\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.348713 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.348672 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-token\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.348713 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.348698 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-credential-socket\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.348974 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.348714 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c7bbe4-d255-480e-a031-b9987c89f47b-istiod-ca-cert\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.348974 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.348750 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-data\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.348974 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.348778 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-podinfo\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.349128 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.349050 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-workload-certs\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.349185 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.349143 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-data\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.349302 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.349277 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-workload-socket\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.349369 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.349317 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-credential-socket\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.349494 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.349473 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c7bbe4-d255-480e-a031-b9987c89f47b-istiod-ca-cert\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.350859 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.350838 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-envoy\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.351100 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.351067 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-podinfo\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.406852 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.406821 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-token\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.407031 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.407008 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g6rg\" (UniqueName: \"kubernetes.io/projected/94c7bbe4-d255-480e-a031-b9987c89f47b-kube-api-access-8g6rg\") pod \"e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.505387 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.505342 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:41.933222 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.933144 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd"] Nov 04 01:32:41.938943 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:32:41.938912 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c7bbe4_d255_480e_a031_b9987c89f47b.slice/crio-53b8a554190696fee321f6340b3e1939e61df9d718fc401f9008058dd58d5e46 WatchSource:0}: Error finding container 53b8a554190696fee321f6340b3e1939e61df9d718fc401f9008058dd58d5e46: Status 404 returned error can't find the container with id 53b8a554190696fee321f6340b3e1939e61df9d718fc401f9008058dd58d5e46 Nov 04 01:32:41.941239 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.941204 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:32:41.941346 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.941285 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:32:41.941346 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:41.941332 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:32:42.184086 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:42.183982 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" event={"ID":"94c7bbe4-d255-480e-a031-b9987c89f47b","Type":"ContainerStarted","Data":"b4c8d845430668ab30a246325fe3bba4f09ada557eae8b020af5febd82a195d1"} Nov 04 01:32:42.184086 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:42.184054 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" event={"ID":"94c7bbe4-d255-480e-a031-b9987c89f47b","Type":"ContainerStarted","Data":"53b8a554190696fee321f6340b3e1939e61df9d718fc401f9008058dd58d5e46"} Nov 04 01:32:42.307436 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:42.306987 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" podStartSLOduration=1.306968028 podStartE2EDuration="1.306968028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:42.302500911 +0000 UTC m=+1312.334234301" watchObservedRunningTime="2026-11-04 01:32:42.306968028 +0000 UTC m=+1312.338701420" Nov 04 01:32:42.506479 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:42.506443 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:42.508207 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:42.508176 5995 patch_prober.go:29] interesting pod/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:32:42.508357 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:42.508229 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" podUID="94c7bbe4-d255-480e-a031-b9987c89f47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 04 01:32:42.939811 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:42.939727 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6"] Nov 04 01:32:42.943476 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:42.943455 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.008512 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.008484 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-cae0ab-openshift-default-dockercfg-ffhqp\"" Nov 04 01:32:43.021138 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.021044 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6"] Nov 04 01:32:43.085918 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.085880 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-envoy\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.086202 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.086172 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-workload-certs\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.086445 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.086391 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-podinfo\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.086587 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.086572 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-workload-socket\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.086677 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.086663 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-data\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.086786 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.086771 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-credential-socket\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.086935 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.086922 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-token\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.090561 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.090529 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istiod-ca-cert\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.090768 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.090747 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj7xw\" (UniqueName: \"kubernetes.io/projected/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-kube-api-access-jj7xw\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.227441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.217048 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-envoy\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.227441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.217099 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-workload-certs\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.227441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.217186 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-podinfo\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.227441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.217227 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-workload-socket\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.227441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.217252 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-data\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.227441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.217284 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-credential-socket\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.227441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.217333 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-token\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.227441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.217381 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istiod-ca-cert\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.227441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.217456 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj7xw\" (UniqueName: \"kubernetes.io/projected/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-kube-api-access-jj7xw\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.227441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.218591 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-data\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.227441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.218834 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-workload-certs\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.227441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.225207 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-credential-socket\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.227441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.225804 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istiod-ca-cert\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.227441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.225884 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-workload-socket\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.227441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.226430 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-podinfo\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.229019 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.228994 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-envoy\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.297129 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.297078 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj7xw\" (UniqueName: \"kubernetes.io/projected/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-kube-api-access-jj7xw\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.297863 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.297799 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-token\") pod \"e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:43.510340 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.510251 5995 patch_prober.go:29] interesting pod/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:32:43.510930 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.510896 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" podUID="94c7bbe4-d255-480e-a031-b9987c89f47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 04 01:32:43.556237 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:43.556188 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:44.017250 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:44.017211 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6"] Nov 04 01:32:44.023341 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:32:44.023293 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9f1fa2_1c99_4c07_ab88_aafd1f5d5bb2.slice/crio-63517e5c8640b9975051393847dcdc89a82bf058c7830b471f4fec5a3100bf9d WatchSource:0}: Error finding container 63517e5c8640b9975051393847dcdc89a82bf058c7830b471f4fec5a3100bf9d: Status 404 returned error can't find the container with id 63517e5c8640b9975051393847dcdc89a82bf058c7830b471f4fec5a3100bf9d Nov 04 01:32:44.025783 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:44.025750 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:32:44.025894 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:44.025834 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:32:44.025894 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:44.025880 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:32:44.222640 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:44.217460 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" event={"ID":"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2","Type":"ContainerStarted","Data":"ec5a5dc4c591c23e8a1a88868d03ed8a7d41a8a797cd6cd220364f5bb701be09"} Nov 04 01:32:44.222640 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:44.217528 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" event={"ID":"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2","Type":"ContainerStarted","Data":"63517e5c8640b9975051393847dcdc89a82bf058c7830b471f4fec5a3100bf9d"} Nov 04 01:32:44.341197 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:44.340734 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" podStartSLOduration=2.340717351 podStartE2EDuration="2.340717351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:44.340109035 +0000 UTC m=+1314.371842427" watchObservedRunningTime="2026-11-04 01:32:44.340717351 +0000 UTC m=+1314.372450743" Nov 04 01:32:44.515461 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:44.513134 5995 patch_prober.go:29] interesting pod/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:32:44.515461 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:44.513191 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" podUID="94c7bbe4-d255-480e-a031-b9987c89f47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 04 01:32:44.556569 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:44.556536 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:45.505996 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:45.505959 5995 patch_prober.go:29] interesting pod/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:32:45.506255 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:45.506017 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" podUID="94c7bbe4-d255-480e-a031-b9987c89f47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 04 01:32:45.575439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:45.570726 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:46.228151 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:46.228120 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:46.229052 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:46.229032 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:32:46.510832 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:46.509505 5995 patch_prober.go:29] interesting pod/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:32:46.510832 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:46.509591 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" podUID="94c7bbe4-d255-480e-a031-b9987c89f47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 04 01:32:47.506032 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:47.505986 5995 patch_prober.go:29] interesting pod/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:32:47.506570 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:47.506055 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" podUID="94c7bbe4-d255-480e-a031-b9987c89f47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 04 01:32:48.506160 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:48.506117 5995 patch_prober.go:29] interesting pod/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 04 01:32:48.506634 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:48.506176 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" podUID="94c7bbe4-d255-480e-a031-b9987c89f47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 04 01:32:49.511042 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:49.511006 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:49.511458 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:49.511429 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:49.512207 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:49.512186 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:32:49.642996 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:32:49.642958 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:32:56.884066 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:56.884035 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:32:59.434507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:59.434469 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq"] Nov 04 01:32:59.438224 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:59.438194 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq" Nov 04 01:32:59.525742 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:59.525711 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-cae0ab\"" Nov 04 01:32:59.538539 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:59.538500 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fce5c04-9c5c-42f5-bfda-e94dbae0b266-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq\" (UID: \"9fce5c04-9c5c-42f5-bfda-e94dbae0b266\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq" Nov 04 01:32:59.538698 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:59.538662 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp8cp\" (UniqueName: \"kubernetes.io/projected/9fce5c04-9c5c-42f5-bfda-e94dbae0b266-kube-api-access-bp8cp\") pod \"payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq\" (UID: \"9fce5c04-9c5c-42f5-bfda-e94dbae0b266\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq" Nov 04 01:32:59.541119 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:59.541098 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq"] Nov 04 01:32:59.647473 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:59.646669 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp8cp\" (UniqueName: \"kubernetes.io/projected/9fce5c04-9c5c-42f5-bfda-e94dbae0b266-kube-api-access-bp8cp\") pod \"payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq\" (UID: \"9fce5c04-9c5c-42f5-bfda-e94dbae0b266\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq" Nov 04 01:32:59.647473 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:59.646842 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fce5c04-9c5c-42f5-bfda-e94dbae0b266-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq\" (UID: \"9fce5c04-9c5c-42f5-bfda-e94dbae0b266\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq" Nov 04 01:32:59.648074 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:59.647983 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fce5c04-9c5c-42f5-bfda-e94dbae0b266-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq\" (UID: \"9fce5c04-9c5c-42f5-bfda-e94dbae0b266\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq" Nov 04 01:32:59.705916 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:59.705885 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp8cp\" (UniqueName: \"kubernetes.io/projected/9fce5c04-9c5c-42f5-bfda-e94dbae0b266-kube-api-access-bp8cp\") pod \"payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq\" (UID: \"9fce5c04-9c5c-42f5-bfda-e94dbae0b266\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq" Nov 04 01:32:59.751164 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:32:59.751129 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq" Nov 04 01:33:00.128714 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:33:00.128681 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fce5c04_9c5c_42f5_bfda_e94dbae0b266.slice/crio-e99904f1bd1a9cd7ad368c1fee3271c91e1319e699f3740a8ba4e3442325e4dd WatchSource:0}: Error finding container e99904f1bd1a9cd7ad368c1fee3271c91e1319e699f3740a8ba4e3442325e4dd: Status 404 returned error can't find the container with id e99904f1bd1a9cd7ad368c1fee3271c91e1319e699f3740a8ba4e3442325e4dd Nov 04 01:33:00.131666 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:00.131634 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq"] Nov 04 01:33:00.360053 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:00.360009 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq" event={"ID":"9fce5c04-9c5c-42f5-bfda-e94dbae0b266","Type":"ContainerStarted","Data":"5ac3ab0c9707ad0c67bd0cd3a01eb55793d7d64b6f05d254e8141f53f8f3aad3"} Nov 04 01:33:00.360240 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:00.360084 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq" Nov 04 01:33:00.360240 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:00.360102 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq" event={"ID":"9fce5c04-9c5c-42f5-bfda-e94dbae0b266","Type":"ContainerStarted","Data":"e99904f1bd1a9cd7ad368c1fee3271c91e1319e699f3740a8ba4e3442325e4dd"} Nov 04 01:33:00.469745 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:00.469689 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq" podStartSLOduration=1.469671186 podStartE2EDuration="1.469671186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:00.468520724 +0000 UTC m=+1330.500254116" watchObservedRunningTime="2026-11-04 01:33:00.469671186 +0000 UTC m=+1330.501404579" Nov 04 01:33:11.320162 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.320124 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s"] Nov 04 01:33:11.323882 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.323859 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.369893 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.369865 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq" Nov 04 01:33:11.373104 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.373071 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-6db2a7-openshift-default-dockercfg-nq654\"" Nov 04 01:33:11.380605 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.380570 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-envoy\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.380759 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.380625 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-data\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.380759 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.380645 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-workload-socket\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.380759 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.380700 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-token\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.380929 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.380841 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-podinfo\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.380929 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.380898 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-workload-certs\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.381022 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.380967 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0cbdf70-f15f-42bf-b3f0-929212551be1-istiod-ca-cert\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.381095 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.381079 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-credential-socket\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.381206 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.381183 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqgj7\" (UniqueName: \"kubernetes.io/projected/d0cbdf70-f15f-42bf-b3f0-929212551be1-kube-api-access-tqgj7\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.422908 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.422869 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s"] Nov 04 01:33:11.482996 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.482944 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-data\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.483250 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.483227 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-workload-socket\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.483442 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.483404 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-token\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.483587 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.483572 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-podinfo\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.483720 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.483706 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-workload-certs\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.483841 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.483827 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0cbdf70-f15f-42bf-b3f0-929212551be1-istiod-ca-cert\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.484035 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.484000 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-credential-socket\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.484213 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.484199 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqgj7\" (UniqueName: \"kubernetes.io/projected/d0cbdf70-f15f-42bf-b3f0-929212551be1-kube-api-access-tqgj7\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.484327 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.484313 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-envoy\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.548236 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.548204 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-envoy\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.557441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.549750 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-workload-certs\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.560975 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.560931 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-podinfo\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.561482 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.561454 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-credential-socket\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.561716 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.561691 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0cbdf70-f15f-42bf-b3f0-929212551be1-istiod-ca-cert\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.565050 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.564576 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-data\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.573857 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.565291 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-workload-socket\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.573857 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.565935 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqgj7\" (UniqueName: \"kubernetes.io/projected/d0cbdf70-f15f-42bf-b3f0-929212551be1-kube-api-access-tqgj7\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.573857 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.571953 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-token\") pod \"e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:11.637850 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:11.637820 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:12.098722 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:12.098689 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s"] Nov 04 01:33:12.104243 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:33:12.104220 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0cbdf70_f15f_42bf_b3f0_929212551be1.slice/crio-802677cfb932a79ce49c9b8a9a388bc2f463215a89a5d9d18bd16eea69dc1f0d WatchSource:0}: Error finding container 802677cfb932a79ce49c9b8a9a388bc2f463215a89a5d9d18bd16eea69dc1f0d: Status 404 returned error can't find the container with id 802677cfb932a79ce49c9b8a9a388bc2f463215a89a5d9d18bd16eea69dc1f0d Nov 04 01:33:12.106860 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:12.106823 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:33:12.106942 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:12.106898 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:33:12.106942 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:12.106930 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:33:12.437029 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:12.436938 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" event={"ID":"d0cbdf70-f15f-42bf-b3f0-929212551be1","Type":"ContainerStarted","Data":"897c7b70436718d538c4e18c95e1e96c3fe13a02b5b0bb4196f6516823f89b87"} Nov 04 01:33:12.437029 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:12.436995 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" event={"ID":"d0cbdf70-f15f-42bf-b3f0-929212551be1","Type":"ContainerStarted","Data":"802677cfb932a79ce49c9b8a9a388bc2f463215a89a5d9d18bd16eea69dc1f0d"} Nov 04 01:33:12.555056 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:12.555003 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" podStartSLOduration=1.5549881129999998 podStartE2EDuration="1.554988113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:12.553193496 +0000 UTC m=+1342.584926887" watchObservedRunningTime="2026-11-04 01:33:12.554988113 +0000 UTC m=+1342.586721502" Nov 04 01:33:12.638782 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:12.638744 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:13.643633 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:13.643602 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:14.449295 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:14.449262 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:14.450163 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:14.450146 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:33:19.209218 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.209177 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv"] Nov 04 01:33:19.214443 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.214404 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" Nov 04 01:33:19.286571 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.286535 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07dd8e8d-fed9-42fa-a391-dc66bb56a1a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv\" (UID: \"07dd8e8d-fed9-42fa-a391-dc66bb56a1a2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" Nov 04 01:33:19.287019 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.286998 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjhg8\" (UniqueName: \"kubernetes.io/projected/07dd8e8d-fed9-42fa-a391-dc66bb56a1a2-kube-api-access-tjhg8\") pod \"payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv\" (UID: \"07dd8e8d-fed9-42fa-a391-dc66bb56a1a2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" Nov 04 01:33:19.336437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.321355 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-6db2a7\"" Nov 04 01:33:19.336437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.324598 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-6db2a7-dockercfg-nwtqk\"" Nov 04 01:33:19.336437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.330338 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv"] Nov 04 01:33:19.382440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.378510 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm"] Nov 04 01:33:19.388854 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.388815 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07dd8e8d-fed9-42fa-a391-dc66bb56a1a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv\" (UID: \"07dd8e8d-fed9-42fa-a391-dc66bb56a1a2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" Nov 04 01:33:19.389267 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.389243 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjhg8\" (UniqueName: \"kubernetes.io/projected/07dd8e8d-fed9-42fa-a391-dc66bb56a1a2-kube-api-access-tjhg8\") pod \"payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv\" (UID: \"07dd8e8d-fed9-42fa-a391-dc66bb56a1a2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" Nov 04 01:33:19.390202 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.390171 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07dd8e8d-fed9-42fa-a391-dc66bb56a1a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv\" (UID: \"07dd8e8d-fed9-42fa-a391-dc66bb56a1a2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" Nov 04 01:33:19.391156 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.391135 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" Nov 04 01:33:19.411849 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.411818 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm"] Nov 04 01:33:19.447309 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.447272 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjhg8\" (UniqueName: \"kubernetes.io/projected/07dd8e8d-fed9-42fa-a391-dc66bb56a1a2-kube-api-access-tjhg8\") pod \"payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv\" (UID: \"07dd8e8d-fed9-42fa-a391-dc66bb56a1a2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" Nov 04 01:33:19.490781 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.490691 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-str2s\" (UniqueName: \"kubernetes.io/projected/384eb14f-2381-4803-87fb-9c2a6ed88529-kube-api-access-str2s\") pod \"payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm\" (UID: \"384eb14f-2381-4803-87fb-9c2a6ed88529\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" Nov 04 01:33:19.490938 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.490832 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/384eb14f-2381-4803-87fb-9c2a6ed88529-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm\" (UID: \"384eb14f-2381-4803-87fb-9c2a6ed88529\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" Nov 04 01:33:19.528995 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.528961 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" Nov 04 01:33:19.591749 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.591720 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/384eb14f-2381-4803-87fb-9c2a6ed88529-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm\" (UID: \"384eb14f-2381-4803-87fb-9c2a6ed88529\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" Nov 04 01:33:19.591889 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.591873 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-str2s\" (UniqueName: \"kubernetes.io/projected/384eb14f-2381-4803-87fb-9c2a6ed88529-kube-api-access-str2s\") pod \"payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm\" (UID: \"384eb14f-2381-4803-87fb-9c2a6ed88529\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" Nov 04 01:33:19.593038 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.593006 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/384eb14f-2381-4803-87fb-9c2a6ed88529-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm\" (UID: \"384eb14f-2381-4803-87fb-9c2a6ed88529\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" Nov 04 01:33:19.647670 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:33:19.647605 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:33:19.649892 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.649861 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-str2s\" (UniqueName: \"kubernetes.io/projected/384eb14f-2381-4803-87fb-9c2a6ed88529-kube-api-access-str2s\") pod \"payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm\" (UID: \"384eb14f-2381-4803-87fb-9c2a6ed88529\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" Nov 04 01:33:19.710293 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.710247 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" Nov 04 01:33:19.943975 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:19.943937 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv"] Nov 04 01:33:19.950336 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:33:19.950305 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07dd8e8d_fed9_42fa_a391_dc66bb56a1a2.slice/crio-2452e8d91bde2aec573891e8999458a5a96542044b02c1ce7335c820bd0aac68 WatchSource:0}: Error finding container 2452e8d91bde2aec573891e8999458a5a96542044b02c1ce7335c820bd0aac68: Status 404 returned error can't find the container with id 2452e8d91bde2aec573891e8999458a5a96542044b02c1ce7335c820bd0aac68 Nov 04 01:33:20.089772 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:20.089740 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm"] Nov 04 01:33:20.096498 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:33:20.096467 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384eb14f_2381_4803_87fb_9c2a6ed88529.slice/crio-ed141acc873a1969e69b4a5fb955c59d4b815e15956034870c6f65f9c3b4b63c WatchSource:0}: Error finding container ed141acc873a1969e69b4a5fb955c59d4b815e15956034870c6f65f9c3b4b63c: Status 404 returned error can't find the container with id ed141acc873a1969e69b4a5fb955c59d4b815e15956034870c6f65f9c3b4b63c Nov 04 01:33:20.489157 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:20.489116 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" event={"ID":"384eb14f-2381-4803-87fb-9c2a6ed88529","Type":"ContainerStarted","Data":"c0098b4bb062760369f3a2c57021aed792dcc766fd28e986fe8c8883b244d64f"} Nov 04 01:33:20.489603 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:20.489181 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" event={"ID":"384eb14f-2381-4803-87fb-9c2a6ed88529","Type":"ContainerStarted","Data":"ed141acc873a1969e69b4a5fb955c59d4b815e15956034870c6f65f9c3b4b63c"} Nov 04 01:33:20.489603 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:20.489225 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" Nov 04 01:33:20.491482 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:20.491455 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" event={"ID":"07dd8e8d-fed9-42fa-a391-dc66bb56a1a2","Type":"ContainerStarted","Data":"4f4739090cde7fcc25b024fb9bc167d5a3d1685b7c70880e26cf82b24644655a"} Nov 04 01:33:20.491628 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:20.491496 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" event={"ID":"07dd8e8d-fed9-42fa-a391-dc66bb56a1a2","Type":"ContainerStarted","Data":"2452e8d91bde2aec573891e8999458a5a96542044b02c1ce7335c820bd0aac68"} Nov 04 01:33:20.491628 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:20.491560 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" Nov 04 01:33:20.600599 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:20.600544 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" podStartSLOduration=1.600528611 podStartE2EDuration="1.600528611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:20.598316426 +0000 UTC m=+1350.630049816" watchObservedRunningTime="2026-11-04 01:33:20.600528611 +0000 UTC m=+1350.632262051" Nov 04 01:33:20.714103 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:20.712841 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" podStartSLOduration=1.7128225769999998 podStartE2EDuration="1.712822577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:20.709070615 +0000 UTC m=+1350.740804006" watchObservedRunningTime="2026-11-04 01:33:20.712822577 +0000 UTC m=+1350.744555998" Nov 04 01:33:21.596615 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:21.596583 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:33:29.054035 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:29.053935 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm"] Nov 04 01:33:29.054580 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:29.054266 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" podUID="7ca12a66-6407-4747-967c-7e31aadc46d2" containerName="istio-proxy" containerID="cri-o://7f49465316a4b373ff681c8186364c3fce16acf457db45d27e516120640df50a" gracePeriod=30 Nov 04 01:33:31.501559 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:31.501529 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" Nov 04 01:33:31.501972 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:31.501596 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" Nov 04 01:33:33.274815 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:33.274776 5995 patch_prober.go:29] interesting pod/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:33:33.275204 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:33.274843 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" podUID="7ca12a66-6407-4747-967c-7e31aadc46d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:33:34.319090 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.319021 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:33:34.482031 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.481997 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-workload-certs\") pod \"7ca12a66-6407-4747-967c-7e31aadc46d2\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " Nov 04 01:33:34.482236 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.482102 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-data\") pod \"7ca12a66-6407-4747-967c-7e31aadc46d2\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " Nov 04 01:33:34.482236 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.482136 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-podinfo\") pod \"7ca12a66-6407-4747-967c-7e31aadc46d2\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " Nov 04 01:33:34.482236 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.482169 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-credential-socket\") pod \"7ca12a66-6407-4747-967c-7e31aadc46d2\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " Nov 04 01:33:34.482391 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.482267 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-token\" (UniqueName: \"kubernetes.io/projected/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-token\") pod \"7ca12a66-6407-4747-967c-7e31aadc46d2\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " Nov 04 01:33:34.482391 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.482331 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-workload-socket\") pod \"7ca12a66-6407-4747-967c-7e31aadc46d2\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " Nov 04 01:33:34.482391 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.482345 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-workload-certs" pod "7ca12a66-6407-4747-967c-7e31aadc46d2" (UID: "7ca12a66-6407-4747-967c-7e31aadc46d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:34.482391 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.482386 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-envoy\") pod \"7ca12a66-6407-4747-967c-7e31aadc46d2\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " Nov 04 01:33:34.482615 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.482465 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca12a66-6407-4747-967c-7e31aadc46d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca12a66-6407-4747-967c-7e31aadc46d2-istiod-ca-cert\") pod \"7ca12a66-6407-4747-967c-7e31aadc46d2\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " Nov 04 01:33:34.482615 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.482543 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca12a66-6407-4747-967c-7e31aadc46d2-kube-api-access-qrqbt\" (UniqueName: \"kubernetes.io/projected/7ca12a66-6407-4747-967c-7e31aadc46d2-kube-api-access-qrqbt\") pod \"7ca12a66-6407-4747-967c-7e31aadc46d2\" (UID: \"7ca12a66-6407-4747-967c-7e31aadc46d2\") " Nov 04 01:33:34.482901 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.482875 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-data" pod "7ca12a66-6407-4747-967c-7e31aadc46d2" (UID: "7ca12a66-6407-4747-967c-7e31aadc46d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:34.483055 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.482963 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.483055 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.482984 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.483055 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.483014 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-workload-socket" pod "7ca12a66-6407-4747-967c-7e31aadc46d2" (UID: "7ca12a66-6407-4747-967c-7e31aadc46d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:34.483374 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.483166 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-credential-socket" pod "7ca12a66-6407-4747-967c-7e31aadc46d2" (UID: "7ca12a66-6407-4747-967c-7e31aadc46d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:34.483643 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.483617 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca12a66-6407-4747-967c-7e31aadc46d2-istiod-ca-cert" pod "7ca12a66-6407-4747-967c-7e31aadc46d2" (UID: "7ca12a66-6407-4747-967c-7e31aadc46d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:34.485243 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.485201 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-envoy" pod "7ca12a66-6407-4747-967c-7e31aadc46d2" (UID: "7ca12a66-6407-4747-967c-7e31aadc46d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:34.485780 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.485754 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-token" pod "7ca12a66-6407-4747-967c-7e31aadc46d2" (UID: "7ca12a66-6407-4747-967c-7e31aadc46d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:34.485869 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.485827 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca12a66-6407-4747-967c-7e31aadc46d2-kube-api-access-qrqbt" pod "7ca12a66-6407-4747-967c-7e31aadc46d2" (UID: "7ca12a66-6407-4747-967c-7e31aadc46d2"). InnerVolumeSpecName "kube-api-access-qrqbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:34.486509 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.486492 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-podinfo" pod "7ca12a66-6407-4747-967c-7e31aadc46d2" (UID: "7ca12a66-6407-4747-967c-7e31aadc46d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:34.584335 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.584230 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.584335 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.584265 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.584335 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.584281 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.584335 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.584294 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.584335 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.584307 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca12a66-6407-4747-967c-7e31aadc46d2-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.584335 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.584320 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca12a66-6407-4747-967c-7e31aadc46d2-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.584335 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.584333 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrqbt\" (UniqueName: \"kubernetes.io/projected/7ca12a66-6407-4747-967c-7e31aadc46d2-kube-api-access-qrqbt\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.585693 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.585663 5995 generic.go:356] "Generic (PLEG): container finished" podID="7ca12a66-6407-4747-967c-7e31aadc46d2" containerID="7f49465316a4b373ff681c8186364c3fce16acf457db45d27e516120640df50a" exitCode=0 Nov 04 01:33:34.585836 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.585715 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" Nov 04 01:33:34.585836 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.585746 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" event={"ID":"7ca12a66-6407-4747-967c-7e31aadc46d2","Type":"ContainerDied","Data":"7f49465316a4b373ff681c8186364c3fce16acf457db45d27e516120640df50a"} Nov 04 01:33:34.585836 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.585807 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm" event={"ID":"7ca12a66-6407-4747-967c-7e31aadc46d2","Type":"ContainerDied","Data":"d9664ddf4ed250fb5cec03c61b36be10e7bd58e74dbb20fc063d60abff79e58c"} Nov 04 01:33:34.585957 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.585837 5995 scope.go:122] "RemoveContainer" containerID="7f49465316a4b373ff681c8186364c3fce16acf457db45d27e516120640df50a" Nov 04 01:33:34.597248 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.597226 5995 scope.go:122] "RemoveContainer" containerID="c485cea884b487821f37a24d2d2ea5907e50e49887b73bad6591a2312ea3a194" Nov 04 01:33:34.608524 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.608501 5995 scope.go:122] "RemoveContainer" containerID="7f49465316a4b373ff681c8186364c3fce16acf457db45d27e516120640df50a" Nov 04 01:33:34.608863 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:33:34.608838 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f49465316a4b373ff681c8186364c3fce16acf457db45d27e516120640df50a\": container with ID starting with 7f49465316a4b373ff681c8186364c3fce16acf457db45d27e516120640df50a not found: ID does not exist" containerID="7f49465316a4b373ff681c8186364c3fce16acf457db45d27e516120640df50a" Nov 04 01:33:34.608930 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.608877 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f49465316a4b373ff681c8186364c3fce16acf457db45d27e516120640df50a"} err="failed to get container status \"7f49465316a4b373ff681c8186364c3fce16acf457db45d27e516120640df50a\": rpc error: code = NotFound desc = could not find container \"7f49465316a4b373ff681c8186364c3fce16acf457db45d27e516120640df50a\": container with ID starting with 7f49465316a4b373ff681c8186364c3fce16acf457db45d27e516120640df50a not found: ID does not exist" Nov 04 01:33:34.608930 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.608901 5995 scope.go:122] "RemoveContainer" containerID="c485cea884b487821f37a24d2d2ea5907e50e49887b73bad6591a2312ea3a194" Nov 04 01:33:34.609187 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:33:34.609162 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c485cea884b487821f37a24d2d2ea5907e50e49887b73bad6591a2312ea3a194\": container with ID starting with c485cea884b487821f37a24d2d2ea5907e50e49887b73bad6591a2312ea3a194 not found: ID does not exist" containerID="c485cea884b487821f37a24d2d2ea5907e50e49887b73bad6591a2312ea3a194" Nov 04 01:33:34.609238 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.609196 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c485cea884b487821f37a24d2d2ea5907e50e49887b73bad6591a2312ea3a194"} err="failed to get container status \"c485cea884b487821f37a24d2d2ea5907e50e49887b73bad6591a2312ea3a194\": rpc error: code = NotFound desc = could not find container \"c485cea884b487821f37a24d2d2ea5907e50e49887b73bad6591a2312ea3a194\": container with ID starting with c485cea884b487821f37a24d2d2ea5907e50e49887b73bad6591a2312ea3a194 not found: ID does not exist" Nov 04 01:33:34.745401 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.745360 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm"] Nov 04 01:33:34.750120 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:34.750088 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-7fsjm"] Nov 04 01:33:36.130660 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:36.130620 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca12a66-6407-4747-967c-7e31aadc46d2" path="/var/lib/kubelet/pods/7ca12a66-6407-4747-967c-7e31aadc46d2/volumes" Nov 04 01:33:38.361813 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.361758 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw"] Nov 04 01:33:38.362504 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.362487 5995 cpu_manager.go:414] "containerMap: removing container" podUID="7ca12a66-6407-4747-967c-7e31aadc46d2" containerName="istio-proxy" Nov 04 01:33:38.362560 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.362505 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca12a66-6407-4747-967c-7e31aadc46d2" containerName="istio-proxy" Nov 04 01:33:38.362560 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.362542 5995 cpu_manager.go:414] "containerMap: removing container" podUID="7ca12a66-6407-4747-967c-7e31aadc46d2" containerName="istio-proxy" Nov 04 01:33:38.362560 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.362548 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca12a66-6407-4747-967c-7e31aadc46d2" containerName="istio-proxy" Nov 04 01:33:38.362664 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.362654 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca12a66-6407-4747-967c-7e31aadc46d2" containerName="istio-proxy" Nov 04 01:33:38.366306 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.366287 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.456751 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.456709 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw"] Nov 04 01:33:38.457711 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.457679 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-data\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.458127 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.458104 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-credential-socket\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.458300 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.458284 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-token\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.458453 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.458436 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-podinfo\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.458618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.458603 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-workload-certs\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.459931 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.458752 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istiod-ca-cert\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.459931 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.459670 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-workload-socket\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.459931 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.459703 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-envoy\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.459931 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.459726 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d7pz\" (UniqueName: \"kubernetes.io/projected/ae00f727-de4d-4432-8f93-f8ad9d1aae98-kube-api-access-7d7pz\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.472674 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.472639 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-b1047d-openshift-default-dockercfg-fzrhz\"" Nov 04 01:33:38.499045 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.499009 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s"] Nov 04 01:33:38.500969 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.500933 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca12a66-6407-4747-967c-7e31aadc46d2" containerName="istio-proxy" Nov 04 01:33:38.509346 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.509314 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.541284 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.541225 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s"] Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.561487 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-data\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.561582 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-envoy\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.561619 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-token\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.561670 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-podinfo\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.561707 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-data\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.561746 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-credential-socket\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.561771 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-token\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.561799 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-podinfo\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.561832 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-workload-certs\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.561865 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-workload-certs\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.561899 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istiod-ca-cert\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.561927 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45cc4145-4d28-44b1-a76d-a24535f207e7-istiod-ca-cert\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.561981 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psqnt\" (UniqueName: \"kubernetes.io/projected/45cc4145-4d28-44b1-a76d-a24535f207e7-kube-api-access-psqnt\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.562029 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-workload-socket\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.562054 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-envoy\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.562077 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d7pz\" (UniqueName: \"kubernetes.io/projected/ae00f727-de4d-4432-8f93-f8ad9d1aae98-kube-api-access-7d7pz\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.562120 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-credential-socket\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.562222 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-workload-socket\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.562932 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-data\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.563285 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-credential-socket\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.564258 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-workload-certs\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.565315 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istiod-ca-cert\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.565952 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-workload-socket\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.568465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.568266 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-podinfo\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.569941 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.569683 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-envoy\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.624252 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.624153 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-token\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.624446 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.624253 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d7pz\" (UniqueName: \"kubernetes.io/projected/ae00f727-de4d-4432-8f93-f8ad9d1aae98-kube-api-access-7d7pz\") pod \"e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.663521 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.663483 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45cc4145-4d28-44b1-a76d-a24535f207e7-istiod-ca-cert\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.663689 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.663536 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psqnt\" (UniqueName: \"kubernetes.io/projected/45cc4145-4d28-44b1-a76d-a24535f207e7-kube-api-access-psqnt\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.663689 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.663572 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-credential-socket\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.663689 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.663616 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-workload-socket\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.663883 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.663758 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-envoy\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.663883 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.663804 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-token\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.663883 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.663856 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-podinfo\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.664043 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.663893 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-data\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.664043 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.663962 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-workload-certs\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.664043 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.663993 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-workload-socket\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.664155 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.664044 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-credential-socket\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.664319 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.664276 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45cc4145-4d28-44b1-a76d-a24535f207e7-istiod-ca-cert\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.664319 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.664312 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-workload-certs\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.664470 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.664287 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-data\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.665902 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.665884 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-envoy\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.666286 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.666269 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-podinfo\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.680317 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.680287 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:38.725940 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.725903 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-token\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.726138 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.726016 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psqnt\" (UniqueName: \"kubernetes.io/projected/45cc4145-4d28-44b1-a76d-a24535f207e7-kube-api-access-psqnt\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:38.847514 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:38.847470 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:39.122716 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:39.122680 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw"] Nov 04 01:33:39.128211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:33:39.128186 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae00f727_de4d_4432_8f93_f8ad9d1aae98.slice/crio-788998d1a96afbd601cae574b7ee08c549b0707ec67620207c6c886ff0e45282 WatchSource:0}: Error finding container 788998d1a96afbd601cae574b7ee08c549b0707ec67620207c6c886ff0e45282: Status 404 returned error can't find the container with id 788998d1a96afbd601cae574b7ee08c549b0707ec67620207c6c886ff0e45282 Nov 04 01:33:39.134650 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:39.134613 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:33:39.134748 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:39.134687 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:33:39.134748 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:39.134719 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:33:39.227352 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:39.227319 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s"] Nov 04 01:33:39.238701 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:33:39.238670 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45cc4145_4d28_44b1_a76d_a24535f207e7.slice/crio-98bf916bf1ff4e0f2b9a0ba1aa9091616fcbb0c5884a130affecdc6fbe48c95a WatchSource:0}: Error finding container 98bf916bf1ff4e0f2b9a0ba1aa9091616fcbb0c5884a130affecdc6fbe48c95a: Status 404 returned error can't find the container with id 98bf916bf1ff4e0f2b9a0ba1aa9091616fcbb0c5884a130affecdc6fbe48c95a Nov 04 01:33:39.241221 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:39.241188 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:33:39.241311 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:39.241257 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:33:39.241311 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:39.241286 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:33:39.619154 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:39.619120 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" event={"ID":"45cc4145-4d28-44b1-a76d-a24535f207e7","Type":"ContainerStarted","Data":"93fa42310d0c0bc7faec3e47bd3436d33433ad586dc150e272718696fa2eb3a7"} Nov 04 01:33:39.619617 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:39.619181 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" event={"ID":"45cc4145-4d28-44b1-a76d-a24535f207e7","Type":"ContainerStarted","Data":"98bf916bf1ff4e0f2b9a0ba1aa9091616fcbb0c5884a130affecdc6fbe48c95a"} Nov 04 01:33:39.620832 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:39.620807 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" event={"ID":"ae00f727-de4d-4432-8f93-f8ad9d1aae98","Type":"ContainerStarted","Data":"05a3ac89aa680107b02505ac1b0ecb74d14fbae08cca7dbc260ec43ec0a24f05"} Nov 04 01:33:39.620962 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:39.620847 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" event={"ID":"ae00f727-de4d-4432-8f93-f8ad9d1aae98","Type":"ContainerStarted","Data":"788998d1a96afbd601cae574b7ee08c549b0707ec67620207c6c886ff0e45282"} Nov 04 01:33:39.681374 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:39.681346 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:39.734385 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:39.734327 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" podStartSLOduration=1.734309763 podStartE2EDuration="1.734309763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:39.732256366 +0000 UTC m=+1369.763989756" watchObservedRunningTime="2026-11-04 01:33:39.734309763 +0000 UTC m=+1369.766043152" Nov 04 01:33:39.848595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:39.848557 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:39.853711 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:39.853667 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" podStartSLOduration=1.853653416 podStartE2EDuration="1.853653416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:39.852351385 +0000 UTC m=+1369.884084791" watchObservedRunningTime="2026-11-04 01:33:39.853653416 +0000 UTC m=+1369.885386805" Nov 04 01:33:40.693079 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:40.693033 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:40.849858 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:40.849819 5995 patch_prober.go:29] interesting pod/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 04 01:33:40.850027 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:40.849878 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" podUID="45cc4145-4d28-44b1-a76d-a24535f207e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 04 01:33:41.633177 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:41.633141 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:41.634012 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:41.633991 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:33:41.847889 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:41.847855 5995 patch_prober.go:29] interesting pod/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 04 01:33:41.848254 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:41.847914 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" podUID="45cc4145-4d28-44b1-a76d-a24535f207e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 04 01:33:42.848070 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:42.848032 5995 patch_prober.go:29] interesting pod/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 04 01:33:42.848512 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:42.848089 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" podUID="45cc4145-4d28-44b1-a76d-a24535f207e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 04 01:33:43.848240 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:43.848211 5995 patch_prober.go:29] interesting pod/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 04 01:33:43.848684 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:43.848660 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" podUID="45cc4145-4d28-44b1-a76d-a24535f207e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 04 01:33:44.848860 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:44.848828 5995 patch_prober.go:29] interesting pod/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 04 01:33:44.849316 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:44.848881 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" podUID="45cc4145-4d28-44b1-a76d-a24535f207e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 04 01:33:45.848504 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:45.848458 5995 patch_prober.go:29] interesting pod/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 04 01:33:45.848692 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:45.848533 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" podUID="45cc4145-4d28-44b1-a76d-a24535f207e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 04 01:33:46.852137 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:46.852107 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:46.852554 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:46.852531 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:46.853288 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:46.853269 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:33:49.642603 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:33:49.642569 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:33:55.613707 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.613663 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz"] Nov 04 01:33:55.622060 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.622025 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" Nov 04 01:33:55.644682 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.644645 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7650b8ae-725e-470e-8960-5053aa977ea0-plugins-config-volume\") pod \"payload-processing-external-model-6fddb87f65-hxqhz\" (UID: \"7650b8ae-725e-470e-8960-5053aa977ea0\") " pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" Nov 04 01:33:55.644924 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.644719 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v72rq\" (UniqueName: \"kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-kube-api-access-v72rq\") pod \"payload-processing-external-model-6fddb87f65-hxqhz\" (UID: \"7650b8ae-725e-470e-8960-5053aa977ea0\") " pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" Nov 04 01:33:55.645001 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.644928 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials\") pod \"payload-processing-external-model-6fddb87f65-hxqhz\" (UID: \"7650b8ae-725e-470e-8960-5053aa977ea0\") " pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" Nov 04 01:33:55.645067 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.645049 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7650b8ae-725e-470e-8960-5053aa977ea0-routing-overlay-volume\") pod \"payload-processing-external-model-6fddb87f65-hxqhz\" (UID: \"7650b8ae-725e-470e-8960-5053aa977ea0\") " pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" Nov 04 01:33:55.658602 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.658562 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz"] Nov 04 01:33:55.675808 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.675778 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 04 01:33:55.676124 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.675829 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bec17d7cdfd4\"" Nov 04 01:33:55.676235 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.675941 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 04 01:33:55.676332 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.676075 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ca76f974b55b\"" Nov 04 01:33:55.704451 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.702730 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qxd95\"" Nov 04 01:33:55.704451 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.702805 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 04 01:33:55.704451 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.702868 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 04 01:33:55.746202 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.746162 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials\") pod \"payload-processing-external-model-6fddb87f65-hxqhz\" (UID: \"7650b8ae-725e-470e-8960-5053aa977ea0\") " pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" Nov 04 01:33:55.746382 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.746289 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7650b8ae-725e-470e-8960-5053aa977ea0-routing-overlay-volume\") pod \"payload-processing-external-model-6fddb87f65-hxqhz\" (UID: \"7650b8ae-725e-470e-8960-5053aa977ea0\") " pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" Nov 04 01:33:55.746382 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.746374 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7650b8ae-725e-470e-8960-5053aa977ea0-plugins-config-volume\") pod \"payload-processing-external-model-6fddb87f65-hxqhz\" (UID: \"7650b8ae-725e-470e-8960-5053aa977ea0\") " pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" Nov 04 01:33:55.746478 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.746430 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v72rq\" (UniqueName: \"kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-kube-api-access-v72rq\") pod \"payload-processing-external-model-6fddb87f65-hxqhz\" (UID: \"7650b8ae-725e-470e-8960-5053aa977ea0\") " pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" Nov 04 01:33:55.747442 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.747382 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7650b8ae-725e-470e-8960-5053aa977ea0-routing-overlay-volume\") pod \"payload-processing-external-model-6fddb87f65-hxqhz\" (UID: \"7650b8ae-725e-470e-8960-5053aa977ea0\") " pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" Nov 04 01:33:55.747650 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.747473 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7650b8ae-725e-470e-8960-5053aa977ea0-plugins-config-volume\") pod \"payload-processing-external-model-6fddb87f65-hxqhz\" (UID: \"7650b8ae-725e-470e-8960-5053aa977ea0\") " pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" Nov 04 01:33:55.749321 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.749298 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials\") pod \"payload-processing-external-model-6fddb87f65-hxqhz\" (UID: \"7650b8ae-725e-470e-8960-5053aa977ea0\") " pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" Nov 04 01:33:55.804035 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.804004 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v72rq\" (UniqueName: \"kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-kube-api-access-v72rq\") pod \"payload-processing-external-model-6fddb87f65-hxqhz\" (UID: \"7650b8ae-725e-470e-8960-5053aa977ea0\") " pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" Nov 04 01:33:55.938400 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:55.938313 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" Nov 04 01:33:56.322645 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:56.322611 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz"] Nov 04 01:33:56.326971 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:33:56.326939 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7650b8ae_725e_470e_8960_5053aa977ea0.slice/crio-6cfaaa86df14f466d64b24b986c257e0e9a32dc4395ea90e4e43ace1019fea23 WatchSource:0}: Error finding container 6cfaaa86df14f466d64b24b986c257e0e9a32dc4395ea90e4e43ace1019fea23: Status 404 returned error can't find the container with id 6cfaaa86df14f466d64b24b986c257e0e9a32dc4395ea90e4e43ace1019fea23 Nov 04 01:33:56.759635 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:56.759602 5995 generic.go:356] "Generic (PLEG): container finished" podID="cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" containerID="ec5a5dc4c591c23e8a1a88868d03ed8a7d41a8a797cd6cd220364f5bb701be09" exitCode=0 Nov 04 01:33:56.760068 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:56.759673 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" event={"ID":"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2","Type":"ContainerDied","Data":"ec5a5dc4c591c23e8a1a88868d03ed8a7d41a8a797cd6cd220364f5bb701be09"} Nov 04 01:33:56.760149 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:56.760134 5995 scope.go:122] "RemoveContainer" containerID="ec5a5dc4c591c23e8a1a88868d03ed8a7d41a8a797cd6cd220364f5bb701be09" Nov 04 01:33:56.761729 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:56.761704 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" event={"ID":"7650b8ae-725e-470e-8960-5053aa977ea0","Type":"ContainerStarted","Data":"7646d8e87b4446c538416301b0ef3e0f659653d3c2d75474390acd30f9214199"} Nov 04 01:33:56.761836 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:56.761748 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" event={"ID":"7650b8ae-725e-470e-8960-5053aa977ea0","Type":"ContainerStarted","Data":"6cfaaa86df14f466d64b24b986c257e0e9a32dc4395ea90e4e43ace1019fea23"} Nov 04 01:33:56.761836 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:56.761771 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" Nov 04 01:33:56.761836 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:56.761791 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:33:56.761967 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:56.761867 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:33:56.761967 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:56.761899 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:33:57.030064 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:57.029955 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" podStartSLOduration=2.029937588 podStartE2EDuration="2.029937588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:57.029272643 +0000 UTC m=+1387.061006051" watchObservedRunningTime="2026-11-04 01:33:57.029937588 +0000 UTC m=+1387.061670977" Nov 04 01:33:57.794384 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:57.794342 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" event={"ID":"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2","Type":"ContainerStarted","Data":"3727b1294557a5d235d3e7c5430e36331cb9acfc29fa0eaeeb13e39db390d5f3"} Nov 04 01:33:58.556381 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:58.556346 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:33:58.557821 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:58.557791 5995 patch_prober.go:29] interesting pod/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 04 01:33:58.557935 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:58.557840 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" podUID="cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 04 01:33:59.556862 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:59.556812 5995 patch_prober.go:29] interesting pod/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 04 01:33:59.557286 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:33:59.556883 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" podUID="cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 04 01:34:00.557631 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:00.557589 5995 patch_prober.go:29] interesting pod/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 04 01:34:00.558137 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:00.557657 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" podUID="cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 04 01:34:01.228281 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:01.228245 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:34:01.556794 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:01.556712 5995 patch_prober.go:29] interesting pod/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 04 01:34:01.556794 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:01.556773 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" podUID="cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 04 01:34:02.556778 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:02.556738 5995 patch_prober.go:29] interesting pod/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 04 01:34:02.557302 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:02.556820 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" podUID="cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 04 01:34:03.562102 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:03.562071 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:34:03.563226 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:03.563202 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:34:07.797300 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:07.797272 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" Nov 04 01:34:13.406920 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:13.406889 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:34:19.642915 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:34:19.642874 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:34:23.980173 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:23.980136 5995 generic.go:356] "Generic (PLEG): container finished" podID="d0cbdf70-f15f-42bf-b3f0-929212551be1" containerID="897c7b70436718d538c4e18c95e1e96c3fe13a02b5b0bb4196f6516823f89b87" exitCode=0 Nov 04 01:34:23.980642 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:23.980209 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" event={"ID":"d0cbdf70-f15f-42bf-b3f0-929212551be1","Type":"ContainerDied","Data":"897c7b70436718d538c4e18c95e1e96c3fe13a02b5b0bb4196f6516823f89b87"} Nov 04 01:34:23.980691 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:23.980656 5995 scope.go:122] "RemoveContainer" containerID="897c7b70436718d538c4e18c95e1e96c3fe13a02b5b0bb4196f6516823f89b87" Nov 04 01:34:23.982221 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:23.982189 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:34:23.982336 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:23.982262 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:34:23.982336 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:23.982295 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:34:24.990242 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:24.990208 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" event={"ID":"d0cbdf70-f15f-42bf-b3f0-929212551be1","Type":"ContainerStarted","Data":"32ed93e35a841b8b2f673d534fcfc4f5514ce1bbf79435aa90f2256cf31a0167"} Nov 04 01:34:25.640728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:25.638585 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:34:25.642235 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:25.642203 5995 patch_prober.go:29] interesting pod/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:34:25.642518 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:25.642491 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" podUID="d0cbdf70-f15f-42bf-b3f0-929212551be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 01:34:26.639111 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:26.639072 5995 patch_prober.go:29] interesting pod/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:34:26.639658 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:26.639136 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" podUID="d0cbdf70-f15f-42bf-b3f0-929212551be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 01:34:27.639318 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:27.639282 5995 patch_prober.go:29] interesting pod/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:34:27.639775 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:27.639354 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" podUID="d0cbdf70-f15f-42bf-b3f0-929212551be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 01:34:28.638464 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:28.638428 5995 patch_prober.go:29] interesting pod/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:34:28.638636 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:28.638513 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" podUID="d0cbdf70-f15f-42bf-b3f0-929212551be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 01:34:29.449781 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:29.449744 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:34:29.638305 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:29.638256 5995 patch_prober.go:29] interesting pod/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:34:29.638554 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:29.638337 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" podUID="d0cbdf70-f15f-42bf-b3f0-929212551be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 01:34:30.638902 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:30.638864 5995 patch_prober.go:29] interesting pod/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 01:34:30.639307 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:30.638930 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" podUID="d0cbdf70-f15f-42bf-b3f0-929212551be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 01:34:31.643672 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:31.643640 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:34:31.644681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:31.644663 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:34:37.609134 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.609099 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch"] Nov 04 01:34:37.613084 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.613062 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" Nov 04 01:34:37.648609 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.648570 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3443e17f-4370-4c7e-8d10-8a79d491dca4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch\" (UID: \"3443e17f-4370-4c7e-8d10-8a79d491dca4\") " pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" Nov 04 01:34:37.648784 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.648711 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm4m5\" (UniqueName: \"kubernetes.io/projected/3443e17f-4370-4c7e-8d10-8a79d491dca4-kube-api-access-qm4m5\") pod \"payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch\" (UID: \"3443e17f-4370-4c7e-8d10-8a79d491dca4\") " pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" Nov 04 01:34:37.648906 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.648882 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3443e17f-4370-4c7e-8d10-8a79d491dca4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch\" (UID: \"3443e17f-4370-4c7e-8d10-8a79d491dca4\") " pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" Nov 04 01:34:37.648981 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.648934 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3443e17f-4370-4c7e-8d10-8a79d491dca4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch\" (UID: \"3443e17f-4370-4c7e-8d10-8a79d491dca4\") " pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" Nov 04 01:34:37.662795 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.662768 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-cae0ab\"/\"kube-root-ca.crt\"" Nov 04 01:34:37.709356 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.709305 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-cae0ab\"/\"agc-cred-9d6e8d1bee3e\"" Nov 04 01:34:37.709356 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.709315 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-cae0ab\"/\"routing-overlay\"" Nov 04 01:34:37.709356 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.709348 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-cae0ab\"/\"openshift-service-ca.crt\"" Nov 04 01:34:37.709703 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.709433 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-cae0ab\"/\"payload-processing-external-model-e2e--913b1778-dockercfg-nrxk8\"" Nov 04 01:34:37.709703 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.709505 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-cae0ab\"/\"payload-processing-external-model-plugins-e2e-extmod-b-cae0ab\"" Nov 04 01:34:37.712473 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.712448 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch"] Nov 04 01:34:37.750600 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.750566 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3443e17f-4370-4c7e-8d10-8a79d491dca4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch\" (UID: \"3443e17f-4370-4c7e-8d10-8a79d491dca4\") " pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" Nov 04 01:34:37.750773 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.750664 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm4m5\" (UniqueName: \"kubernetes.io/projected/3443e17f-4370-4c7e-8d10-8a79d491dca4-kube-api-access-qm4m5\") pod \"payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch\" (UID: \"3443e17f-4370-4c7e-8d10-8a79d491dca4\") " pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" Nov 04 01:34:37.750773 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.750753 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3443e17f-4370-4c7e-8d10-8a79d491dca4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch\" (UID: \"3443e17f-4370-4c7e-8d10-8a79d491dca4\") " pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" Nov 04 01:34:37.750854 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.750779 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3443e17f-4370-4c7e-8d10-8a79d491dca4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch\" (UID: \"3443e17f-4370-4c7e-8d10-8a79d491dca4\") " pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" Nov 04 01:34:37.751467 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.751442 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3443e17f-4370-4c7e-8d10-8a79d491dca4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch\" (UID: \"3443e17f-4370-4c7e-8d10-8a79d491dca4\") " pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" Nov 04 01:34:37.751590 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.751553 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3443e17f-4370-4c7e-8d10-8a79d491dca4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch\" (UID: \"3443e17f-4370-4c7e-8d10-8a79d491dca4\") " pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" Nov 04 01:34:37.753330 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.753305 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3443e17f-4370-4c7e-8d10-8a79d491dca4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch\" (UID: \"3443e17f-4370-4c7e-8d10-8a79d491dca4\") " pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" Nov 04 01:34:37.808800 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.808774 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm4m5\" (UniqueName: \"kubernetes.io/projected/3443e17f-4370-4c7e-8d10-8a79d491dca4-kube-api-access-qm4m5\") pod \"payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch\" (UID: \"3443e17f-4370-4c7e-8d10-8a79d491dca4\") " pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" Nov 04 01:34:37.927455 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:37.927335 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" Nov 04 01:34:38.300613 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:38.300576 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch"] Nov 04 01:34:38.306279 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:34:38.306240 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3443e17f_4370_4c7e_8d10_8a79d491dca4.slice/crio-9f5a34038446e1ba052f401fcf89f65cc133c51d310de342e15f66c458da1ba6 WatchSource:0}: Error finding container 9f5a34038446e1ba052f401fcf89f65cc133c51d310de342e15f66c458da1ba6: Status 404 returned error can't find the container with id 9f5a34038446e1ba052f401fcf89f65cc133c51d310de342e15f66c458da1ba6 Nov 04 01:34:39.112921 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:39.112885 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" event={"ID":"3443e17f-4370-4c7e-8d10-8a79d491dca4","Type":"ContainerStarted","Data":"145070b458cb9dd8be2cb021cebcc84fb721896fce134d316f8b50a4b36c05d4"} Nov 04 01:34:39.113348 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:39.112985 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" event={"ID":"3443e17f-4370-4c7e-8d10-8a79d491dca4","Type":"ContainerStarted","Data":"9f5a34038446e1ba052f401fcf89f65cc133c51d310de342e15f66c458da1ba6"} Nov 04 01:34:39.113348 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:39.113017 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" Nov 04 01:34:39.225590 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:39.225541 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" podStartSLOduration=2.225524954 podStartE2EDuration="2.225524954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:34:39.223460175 +0000 UTC m=+1429.255193566" watchObservedRunningTime="2026-11-04 01:34:39.225524954 +0000 UTC m=+1429.257258362" Nov 04 01:34:42.467993 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.467955 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9"] Nov 04 01:34:42.472950 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.472923 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.509767 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.509729 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.509767 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.509768 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.510031 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.509791 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.510031 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.509939 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91871ffd-c8d2-483a-bfc4-5261b89961aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.510031 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.509987 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.510206 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.510082 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.510206 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.510129 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.510206 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.510150 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.510365 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.510266 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2d8c\" (UniqueName: \"kubernetes.io/projected/91871ffd-c8d2-483a-bfc4-5261b89961aa-kube-api-access-c2d8c\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.611926 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.611892 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2d8c\" (UniqueName: \"kubernetes.io/projected/91871ffd-c8d2-483a-bfc4-5261b89961aa-kube-api-access-c2d8c\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.612110 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.611950 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.612110 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.611976 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.612110 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.611994 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.612110 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.612052 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91871ffd-c8d2-483a-bfc4-5261b89961aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.612110 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.612098 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.612394 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.612172 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.612394 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.612235 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.612394 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.612264 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.612565 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.612531 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.612807 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.612747 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.612807 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.612789 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.612980 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.612820 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.613214 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.613194 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91871ffd-c8d2-483a-bfc4-5261b89961aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.614554 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.614524 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.614832 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.614810 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.669804 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.669771 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.670046 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.669994 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2d8c\" (UniqueName: \"kubernetes.io/projected/91871ffd-c8d2-483a-bfc4-5261b89961aa-kube-api-access-c2d8c\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dpdg9\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:42.674347 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.674320 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9"] Nov 04 01:34:42.788127 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:42.788029 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:43.219944 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:43.219900 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9"] Nov 04 01:34:43.224867 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:34:43.224838 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91871ffd_c8d2_483a_bfc4_5261b89961aa.slice/crio-668f4b984584ffecd81ed44a23bc1797c376bc38b65c85b1599252d7b12c24c0 WatchSource:0}: Error finding container 668f4b984584ffecd81ed44a23bc1797c376bc38b65c85b1599252d7b12c24c0: Status 404 returned error can't find the container with id 668f4b984584ffecd81ed44a23bc1797c376bc38b65c85b1599252d7b12c24c0 Nov 04 01:34:43.227379 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:43.227347 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:34:43.227482 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:43.227432 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:34:43.227482 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:43.227465 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:34:44.146030 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:44.145992 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" event={"ID":"91871ffd-c8d2-483a-bfc4-5261b89961aa","Type":"ContainerStarted","Data":"661b3774ad155464d9fbafc85f204cdf8da8450faa3a5a44943408ff5c030189"} Nov 04 01:34:44.146514 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:44.146044 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" event={"ID":"91871ffd-c8d2-483a-bfc4-5261b89961aa","Type":"ContainerStarted","Data":"668f4b984584ffecd81ed44a23bc1797c376bc38b65c85b1599252d7b12c24c0"} Nov 04 01:34:44.352142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:44.352087 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" podStartSLOduration=2.352073327 podStartE2EDuration="2.352073327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:34:44.305399505 +0000 UTC m=+1434.337132895" watchObservedRunningTime="2026-11-04 01:34:44.352073327 +0000 UTC m=+1434.383806707" Nov 04 01:34:44.788982 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:44.788942 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:44.790701 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:44.790672 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:34:44.790824 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:44.790727 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" podUID="91871ffd-c8d2-483a-bfc4-5261b89961aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 04 01:34:45.789002 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:45.788958 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:34:45.789558 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:45.789028 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" podUID="91871ffd-c8d2-483a-bfc4-5261b89961aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 04 01:34:46.788573 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:46.788531 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:34:46.788765 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:46.788600 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" podUID="91871ffd-c8d2-483a-bfc4-5261b89961aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 04 01:34:47.792903 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:47.792861 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:34:47.793332 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:47.792934 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" podUID="91871ffd-c8d2-483a-bfc4-5261b89961aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 04 01:34:48.789449 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:48.788820 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:34:48.789449 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:48.788888 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" podUID="91871ffd-c8d2-483a-bfc4-5261b89961aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 04 01:34:49.642574 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:34:49.642536 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:34:49.789132 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:49.789080 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:34:49.789334 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:49.789153 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" podUID="91871ffd-c8d2-483a-bfc4-5261b89961aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 04 01:34:50.129562 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:50.129536 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" Nov 04 01:34:50.788546 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:50.788502 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:34:50.789038 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:50.788582 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" podUID="91871ffd-c8d2-483a-bfc4-5261b89961aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 04 01:34:51.966891 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:51.966856 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:51.968520 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:51.968496 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:51.969337 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:51.969315 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:34:53.216496 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:34:53.216459 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:35:03.328348 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:03.328313 5995 generic.go:356] "Generic (PLEG): container finished" podID="ae00f727-de4d-4432-8f93-f8ad9d1aae98" containerID="05a3ac89aa680107b02505ac1b0ecb74d14fbae08cca7dbc260ec43ec0a24f05" exitCode=0 Nov 04 01:35:03.328764 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:03.328394 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" event={"ID":"ae00f727-de4d-4432-8f93-f8ad9d1aae98","Type":"ContainerDied","Data":"05a3ac89aa680107b02505ac1b0ecb74d14fbae08cca7dbc260ec43ec0a24f05"} Nov 04 01:35:03.328809 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:03.328793 5995 scope.go:122] "RemoveContainer" containerID="05a3ac89aa680107b02505ac1b0ecb74d14fbae08cca7dbc260ec43ec0a24f05" Nov 04 01:35:03.330219 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:03.330183 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:35:03.330299 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:03.330274 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:35:03.330344 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:03.330320 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:35:04.337667 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:04.337631 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" event={"ID":"ae00f727-de4d-4432-8f93-f8ad9d1aae98","Type":"ContainerStarted","Data":"960df09cfd90851126a1e45e20522c87c22025f6988a79924eca6ebbf5d36ad4"} Nov 04 01:35:04.682917 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:04.682822 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:35:04.683246 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:04.683223 5995 patch_prober.go:29] interesting pod/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:35:04.683379 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:04.683357 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" podUID="ae00f727-de4d-4432-8f93-f8ad9d1aae98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:35:05.683439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:05.680787 5995 patch_prober.go:29] interesting pod/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:35:05.683439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:05.680851 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" podUID="ae00f727-de4d-4432-8f93-f8ad9d1aae98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:35:06.686875 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:06.686833 5995 patch_prober.go:29] interesting pod/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:35:06.687482 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:06.687446 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" podUID="ae00f727-de4d-4432-8f93-f8ad9d1aae98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:35:07.681772 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:07.681734 5995 patch_prober.go:29] interesting pod/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:35:07.682073 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:07.682045 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" podUID="ae00f727-de4d-4432-8f93-f8ad9d1aae98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:35:08.682437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:08.681765 5995 patch_prober.go:29] interesting pod/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:35:08.682437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:08.681841 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" podUID="ae00f727-de4d-4432-8f93-f8ad9d1aae98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 04 01:35:09.685654 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:09.685623 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:35:09.686101 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:09.686050 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:35:09.686756 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:09.686739 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:35:18.093580 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.093537 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82"] Nov 04 01:35:18.097588 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.097565 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82" Nov 04 01:35:18.127989 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.127952 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54eca79d-c7bc-45b1-a815-119efc0f375e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82\" (UID: \"54eca79d-c7bc-45b1-a815-119efc0f375e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82" Nov 04 01:35:18.128157 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.128134 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbzwn\" (UniqueName: \"kubernetes.io/projected/54eca79d-c7bc-45b1-a815-119efc0f375e-kube-api-access-pbzwn\") pod \"payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82\" (UID: \"54eca79d-c7bc-45b1-a815-119efc0f375e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82" Nov 04 01:35:18.146992 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.146962 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d057d7\"" Nov 04 01:35:18.197776 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.197725 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82"] Nov 04 01:35:18.229967 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.229928 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54eca79d-c7bc-45b1-a815-119efc0f375e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82\" (UID: \"54eca79d-c7bc-45b1-a815-119efc0f375e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82" Nov 04 01:35:18.231443 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.231395 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbzwn\" (UniqueName: \"kubernetes.io/projected/54eca79d-c7bc-45b1-a815-119efc0f375e-kube-api-access-pbzwn\") pod \"payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82\" (UID: \"54eca79d-c7bc-45b1-a815-119efc0f375e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82" Nov 04 01:35:18.231751 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.231199 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54eca79d-c7bc-45b1-a815-119efc0f375e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82\" (UID: \"54eca79d-c7bc-45b1-a815-119efc0f375e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82" Nov 04 01:35:18.286436 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.286207 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r"] Nov 04 01:35:18.323460 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.319456 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbzwn\" (UniqueName: \"kubernetes.io/projected/54eca79d-c7bc-45b1-a815-119efc0f375e-kube-api-access-pbzwn\") pod \"payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82\" (UID: \"54eca79d-c7bc-45b1-a815-119efc0f375e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82" Nov 04 01:35:18.368721 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.368624 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r"] Nov 04 01:35:18.369017 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.369002 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r" Nov 04 01:35:18.409971 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.409942 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82" Nov 04 01:35:18.418734 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.418701 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d057d7-dockercfg-hnj7p\"" Nov 04 01:35:18.444507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.444473 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e910c7-62fc-43a4-a513-847a2b968f65-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r\" (UID: \"43e910c7-62fc-43a4-a513-847a2b968f65\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r" Nov 04 01:35:18.444696 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.444682 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stvts\" (UniqueName: \"kubernetes.io/projected/43e910c7-62fc-43a4-a513-847a2b968f65-kube-api-access-stvts\") pod \"payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r\" (UID: \"43e910c7-62fc-43a4-a513-847a2b968f65\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r" Nov 04 01:35:18.545950 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.545913 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stvts\" (UniqueName: \"kubernetes.io/projected/43e910c7-62fc-43a4-a513-847a2b968f65-kube-api-access-stvts\") pod \"payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r\" (UID: \"43e910c7-62fc-43a4-a513-847a2b968f65\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r" Nov 04 01:35:18.546143 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.546011 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e910c7-62fc-43a4-a513-847a2b968f65-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r\" (UID: \"43e910c7-62fc-43a4-a513-847a2b968f65\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r" Nov 04 01:35:18.546855 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.546830 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e910c7-62fc-43a4-a513-847a2b968f65-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r\" (UID: \"43e910c7-62fc-43a4-a513-847a2b968f65\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r" Nov 04 01:35:18.603972 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.603939 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stvts\" (UniqueName: \"kubernetes.io/projected/43e910c7-62fc-43a4-a513-847a2b968f65-kube-api-access-stvts\") pod \"payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r\" (UID: \"43e910c7-62fc-43a4-a513-847a2b968f65\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r" Nov 04 01:35:18.703042 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.703011 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r" Nov 04 01:35:18.790753 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:18.790714 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82"] Nov 04 01:35:18.796360 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:35:18.796332 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54eca79d_c7bc_45b1_a815_119efc0f375e.slice/crio-f14f00e6b47d7a72838a7d1a3d09ab5299e317b3611eb9d1885110069f5776c1 WatchSource:0}: Error finding container f14f00e6b47d7a72838a7d1a3d09ab5299e317b3611eb9d1885110069f5776c1: Status 404 returned error can't find the container with id f14f00e6b47d7a72838a7d1a3d09ab5299e317b3611eb9d1885110069f5776c1 Nov 04 01:35:19.079338 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:19.079250 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r"] Nov 04 01:35:19.084477 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:35:19.084444 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e910c7_62fc_43a4_a513_847a2b968f65.slice/crio-766ea1f6739ec18f0a03b8139cb6f24d25f147e34523bc5de495b8d811ef4fa3 WatchSource:0}: Error finding container 766ea1f6739ec18f0a03b8139cb6f24d25f147e34523bc5de495b8d811ef4fa3: Status 404 returned error can't find the container with id 766ea1f6739ec18f0a03b8139cb6f24d25f147e34523bc5de495b8d811ef4fa3 Nov 04 01:35:19.464739 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:19.464659 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82" event={"ID":"54eca79d-c7bc-45b1-a815-119efc0f375e","Type":"ContainerStarted","Data":"e93b56ba94eac5614fe11d85fadc42b3cd0ab89ea836c713f53edcc792bafe1e"} Nov 04 01:35:19.464739 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:19.464726 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82" event={"ID":"54eca79d-c7bc-45b1-a815-119efc0f375e","Type":"ContainerStarted","Data":"f14f00e6b47d7a72838a7d1a3d09ab5299e317b3611eb9d1885110069f5776c1"} Nov 04 01:35:19.465235 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:19.464878 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82" Nov 04 01:35:19.466218 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:19.466186 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r" event={"ID":"43e910c7-62fc-43a4-a513-847a2b968f65","Type":"ContainerStarted","Data":"87ef638410a8cdd4cb8ddb63673b58caf0ddbfdd8a1b54fd04ad3036cb214982"} Nov 04 01:35:19.466350 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:19.466229 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r" event={"ID":"43e910c7-62fc-43a4-a513-847a2b968f65","Type":"ContainerStarted","Data":"766ea1f6739ec18f0a03b8139cb6f24d25f147e34523bc5de495b8d811ef4fa3"} Nov 04 01:35:19.466350 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:19.466276 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r" Nov 04 01:35:19.575522 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:19.575465 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82" podStartSLOduration=1.57544978 podStartE2EDuration="1.57544978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:19.573753581 +0000 UTC m=+1469.605486973" watchObservedRunningTime="2026-11-04 01:35:19.57544978 +0000 UTC m=+1469.607183166" Nov 04 01:35:19.643308 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:35:19.643270 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:35:19.683707 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:19.683633 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r" podStartSLOduration=1.683618721 podStartE2EDuration="1.683618721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:19.682104115 +0000 UTC m=+1469.713837505" watchObservedRunningTime="2026-11-04 01:35:19.683618721 +0000 UTC m=+1469.715352110" Nov 04 01:35:24.472236 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:24.472198 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:35:29.487840 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:29.487797 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94"] Nov 04 01:35:29.491819 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:29.491795 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94" Nov 04 01:35:29.543120 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:29.543089 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:35:29.567810 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:29.567779 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7feff85-eaf9-4643-8d8b-35ce61938408-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e33933a5-c6x94\" (UID: \"d7feff85-eaf9-4643-8d8b-35ce61938408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94" Nov 04 01:35:29.567991 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:29.567849 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj76v\" (UniqueName: \"kubernetes.io/projected/d7feff85-eaf9-4643-8d8b-35ce61938408-kube-api-access-xj76v\") pod \"maas-api-revoke-sub-e33933a5-c6x94\" (UID: \"d7feff85-eaf9-4643-8d8b-35ce61938408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94" Nov 04 01:35:29.567991 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:29.567954 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7feff85-eaf9-4643-8d8b-35ce61938408-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e33933a5-c6x94\" (UID: \"d7feff85-eaf9-4643-8d8b-35ce61938408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94" Nov 04 01:35:29.589729 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:29.589695 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w799x\"" Nov 04 01:35:29.589729 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:29.589724 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:35:29.589729 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:29.589712 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:35:29.592686 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:29.592662 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94"] Nov 04 01:35:29.669075 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:29.669026 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7feff85-eaf9-4643-8d8b-35ce61938408-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e33933a5-c6x94\" (UID: \"d7feff85-eaf9-4643-8d8b-35ce61938408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94" Nov 04 01:35:29.669287 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:29.669204 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj76v\" (UniqueName: \"kubernetes.io/projected/d7feff85-eaf9-4643-8d8b-35ce61938408-kube-api-access-xj76v\") pod \"maas-api-revoke-sub-e33933a5-c6x94\" (UID: \"d7feff85-eaf9-4643-8d8b-35ce61938408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94" Nov 04 01:35:29.669360 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:29.669282 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7feff85-eaf9-4643-8d8b-35ce61938408-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e33933a5-c6x94\" (UID: \"d7feff85-eaf9-4643-8d8b-35ce61938408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94" Nov 04 01:35:29.670092 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:29.670070 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7feff85-eaf9-4643-8d8b-35ce61938408-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e33933a5-c6x94\" (UID: \"d7feff85-eaf9-4643-8d8b-35ce61938408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94" Nov 04 01:35:29.671467 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:29.671448 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7feff85-eaf9-4643-8d8b-35ce61938408-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e33933a5-c6x94\" (UID: \"d7feff85-eaf9-4643-8d8b-35ce61938408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94" Nov 04 01:35:29.728192 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:29.728154 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj76v\" (UniqueName: \"kubernetes.io/projected/d7feff85-eaf9-4643-8d8b-35ce61938408-kube-api-access-xj76v\") pod \"maas-api-revoke-sub-e33933a5-c6x94\" (UID: \"d7feff85-eaf9-4643-8d8b-35ce61938408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94" Nov 04 01:35:29.804630 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:29.804533 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94" Nov 04 01:35:30.178487 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:30.178377 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94"] Nov 04 01:35:30.184525 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:35:30.184490 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7feff85_eaf9_4643_8d8b_35ce61938408.slice/crio-d03dffc0e2f4c94b4c283680aa7526074432de437aa8cdcbfd95d06ae74ee051 WatchSource:0}: Error finding container d03dffc0e2f4c94b4c283680aa7526074432de437aa8cdcbfd95d06ae74ee051: Status 404 returned error can't find the container with id d03dffc0e2f4c94b4c283680aa7526074432de437aa8cdcbfd95d06ae74ee051 Nov 04 01:35:30.476702 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:30.476653 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82" Nov 04 01:35:30.476838 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:30.476724 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r" Nov 04 01:35:30.559754 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:30.559721 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94" event={"ID":"d7feff85-eaf9-4643-8d8b-35ce61938408","Type":"ContainerStarted","Data":"48fee41e63f42d8e386efd7ef64a212338df50df126bacd14b547a9337b428e4"} Nov 04 01:35:30.560434 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:30.559782 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94" event={"ID":"d7feff85-eaf9-4643-8d8b-35ce61938408","Type":"ContainerStarted","Data":"d03dffc0e2f4c94b4c283680aa7526074432de437aa8cdcbfd95d06ae74ee051"} Nov 04 01:35:30.805910 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:30.805795 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94" podStartSLOduration=1.805777408 podStartE2EDuration="1.805777408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:30.80305233 +0000 UTC m=+1480.834785720" watchObservedRunningTime="2026-11-04 01:35:30.805777408 +0000 UTC m=+1480.837510798" Nov 04 01:35:31.568592 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:31.568553 5995 generic.go:356] "Generic (PLEG): container finished" podID="d7feff85-eaf9-4643-8d8b-35ce61938408" containerID="48fee41e63f42d8e386efd7ef64a212338df50df126bacd14b547a9337b428e4" exitCode=0 Nov 04 01:35:31.569033 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:31.568641 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94" event={"ID":"d7feff85-eaf9-4643-8d8b-35ce61938408","Type":"ContainerDied","Data":"48fee41e63f42d8e386efd7ef64a212338df50df126bacd14b547a9337b428e4"} Nov 04 01:35:32.735969 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:32.735944 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94" Nov 04 01:35:32.807459 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:32.807402 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7feff85-eaf9-4643-8d8b-35ce61938408-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7feff85-eaf9-4643-8d8b-35ce61938408-maas-api-xks-ca\") pod \"d7feff85-eaf9-4643-8d8b-35ce61938408\" (UID: \"d7feff85-eaf9-4643-8d8b-35ce61938408\") " Nov 04 01:35:32.807667 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:32.807604 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7feff85-eaf9-4643-8d8b-35ce61938408-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7feff85-eaf9-4643-8d8b-35ce61938408-maas-api-openshift-ca\") pod \"d7feff85-eaf9-4643-8d8b-35ce61938408\" (UID: \"d7feff85-eaf9-4643-8d8b-35ce61938408\") " Nov 04 01:35:32.807667 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:32.807641 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7feff85-eaf9-4643-8d8b-35ce61938408-kube-api-access-xj76v\" (UniqueName: \"kubernetes.io/projected/d7feff85-eaf9-4643-8d8b-35ce61938408-kube-api-access-xj76v\") pod \"d7feff85-eaf9-4643-8d8b-35ce61938408\" (UID: \"d7feff85-eaf9-4643-8d8b-35ce61938408\") " Nov 04 01:35:32.807982 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:32.807956 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7feff85-eaf9-4643-8d8b-35ce61938408-maas-api-openshift-ca" pod "d7feff85-eaf9-4643-8d8b-35ce61938408" (UID: "d7feff85-eaf9-4643-8d8b-35ce61938408"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:32.808118 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:32.808105 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7feff85-eaf9-4643-8d8b-35ce61938408-maas-api-openshift-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:32.809558 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:32.809535 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7feff85-eaf9-4643-8d8b-35ce61938408-maas-api-xks-ca" pod "d7feff85-eaf9-4643-8d8b-35ce61938408" (UID: "d7feff85-eaf9-4643-8d8b-35ce61938408"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:32.809909 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:32.809881 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7feff85-eaf9-4643-8d8b-35ce61938408-kube-api-access-xj76v" pod "d7feff85-eaf9-4643-8d8b-35ce61938408" (UID: "d7feff85-eaf9-4643-8d8b-35ce61938408"). InnerVolumeSpecName "kube-api-access-xj76v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:32.910237 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:32.909919 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7feff85-eaf9-4643-8d8b-35ce61938408-maas-api-xks-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:32.910237 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:32.909955 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj76v\" (UniqueName: \"kubernetes.io/projected/d7feff85-eaf9-4643-8d8b-35ce61938408-kube-api-access-xj76v\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:33.583819 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:33.583782 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94" event={"ID":"d7feff85-eaf9-4643-8d8b-35ce61938408","Type":"ContainerDied","Data":"d03dffc0e2f4c94b4c283680aa7526074432de437aa8cdcbfd95d06ae74ee051"} Nov 04 01:35:33.583819 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:33.583814 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94" Nov 04 01:35:33.584021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:33.583843 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d03dffc0e2f4c94b4c283680aa7526074432de437aa8cdcbfd95d06ae74ee051" Nov 04 01:35:44.036285 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.036249 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd"] Nov 04 01:35:44.037168 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.037144 5995 cpu_manager.go:414] "containerMap: removing container" podUID="d7feff85-eaf9-4643-8d8b-35ce61938408" containerName="revoke-keys" Nov 04 01:35:44.037168 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.037168 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7feff85-eaf9-4643-8d8b-35ce61938408" containerName="revoke-keys" Nov 04 01:35:44.037347 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.037316 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7feff85-eaf9-4643-8d8b-35ce61938408" containerName="revoke-keys" Nov 04 01:35:44.039904 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.039884 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd" Nov 04 01:35:44.089044 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.089014 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:35:44.089044 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.089022 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:35:44.158057 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.158024 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:35:44.260542 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.222008 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaafc194-ecb8-46a7-904e-f03823474419-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01fa1c0a-btnvd\" (UID: \"eaafc194-ecb8-46a7-904e-f03823474419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd" Nov 04 01:35:44.260710 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.260215 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w799x\"" Nov 04 01:35:44.260710 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.260644 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mcrk\" (UniqueName: \"kubernetes.io/projected/eaafc194-ecb8-46a7-904e-f03823474419-kube-api-access-9mcrk\") pod \"maas-api-revoke-sub-01fa1c0a-btnvd\" (UID: \"eaafc194-ecb8-46a7-904e-f03823474419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd" Nov 04 01:35:44.260710 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.260685 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaafc194-ecb8-46a7-904e-f03823474419-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01fa1c0a-btnvd\" (UID: \"eaafc194-ecb8-46a7-904e-f03823474419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd" Nov 04 01:35:44.263929 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.263901 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd"] Nov 04 01:35:44.362387 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.362303 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mcrk\" (UniqueName: \"kubernetes.io/projected/eaafc194-ecb8-46a7-904e-f03823474419-kube-api-access-9mcrk\") pod \"maas-api-revoke-sub-01fa1c0a-btnvd\" (UID: \"eaafc194-ecb8-46a7-904e-f03823474419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd" Nov 04 01:35:44.362387 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.362351 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaafc194-ecb8-46a7-904e-f03823474419-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01fa1c0a-btnvd\" (UID: \"eaafc194-ecb8-46a7-904e-f03823474419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd" Nov 04 01:35:44.362691 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.362663 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaafc194-ecb8-46a7-904e-f03823474419-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01fa1c0a-btnvd\" (UID: \"eaafc194-ecb8-46a7-904e-f03823474419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd" Nov 04 01:35:44.363249 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.363226 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaafc194-ecb8-46a7-904e-f03823474419-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01fa1c0a-btnvd\" (UID: \"eaafc194-ecb8-46a7-904e-f03823474419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd" Nov 04 01:35:44.365002 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.364980 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaafc194-ecb8-46a7-904e-f03823474419-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01fa1c0a-btnvd\" (UID: \"eaafc194-ecb8-46a7-904e-f03823474419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd" Nov 04 01:35:44.419946 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.419912 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mcrk\" (UniqueName: \"kubernetes.io/projected/eaafc194-ecb8-46a7-904e-f03823474419-kube-api-access-9mcrk\") pod \"maas-api-revoke-sub-01fa1c0a-btnvd\" (UID: \"eaafc194-ecb8-46a7-904e-f03823474419\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd" Nov 04 01:35:44.653474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:44.653360 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd" Nov 04 01:35:45.030212 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:45.030177 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd"] Nov 04 01:35:45.035812 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:35:45.035779 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaafc194_ecb8_46a7_904e_f03823474419.slice/crio-419f1cba97bddad9c02fb40954c73b732aa29d75c527022c6ccf1c0ec1e88d67 WatchSource:0}: Error finding container 419f1cba97bddad9c02fb40954c73b732aa29d75c527022c6ccf1c0ec1e88d67: Status 404 returned error can't find the container with id 419f1cba97bddad9c02fb40954c73b732aa29d75c527022c6ccf1c0ec1e88d67 Nov 04 01:35:45.697909 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:45.697868 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd" event={"ID":"eaafc194-ecb8-46a7-904e-f03823474419","Type":"ContainerStarted","Data":"a10e051d6eb2310000f4fbf625cda1774de124e0ca05f79c5418527a06d8e816"} Nov 04 01:35:45.698297 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:45.697930 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd" event={"ID":"eaafc194-ecb8-46a7-904e-f03823474419","Type":"ContainerStarted","Data":"419f1cba97bddad9c02fb40954c73b732aa29d75c527022c6ccf1c0ec1e88d67"} Nov 04 01:35:45.808681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:45.808624 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd" podStartSLOduration=2.808609295 podStartE2EDuration="2.808609295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:45.806389064 +0000 UTC m=+1495.838122453" watchObservedRunningTime="2026-11-04 01:35:45.808609295 +0000 UTC m=+1495.840342685" Nov 04 01:35:45.875267 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:45.875231 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm"] Nov 04 01:35:45.878447 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:45.878425 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" Nov 04 01:35:45.937068 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:45.937031 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm"] Nov 04 01:35:45.989653 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:45.989566 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkcf4\" (UniqueName: \"kubernetes.io/projected/c40ae805-9960-4a82-8ba6-b39b6bc6b458-kube-api-access-xkcf4\") pod \"payload-processing-external-model-5dddfc6875-9n4rm\" (UID: \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\") " pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" Nov 04 01:35:45.989653 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:45.989622 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c40ae805-9960-4a82-8ba6-b39b6bc6b458-provider-credentials\") pod \"payload-processing-external-model-5dddfc6875-9n4rm\" (UID: \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\") " pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" Nov 04 01:35:45.989899 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:45.989780 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c40ae805-9960-4a82-8ba6-b39b6bc6b458-plugins-config-volume\") pod \"payload-processing-external-model-5dddfc6875-9n4rm\" (UID: \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\") " pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" Nov 04 01:35:45.990005 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:45.989980 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c40ae805-9960-4a82-8ba6-b39b6bc6b458-routing-overlay-volume\") pod \"payload-processing-external-model-5dddfc6875-9n4rm\" (UID: \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\") " pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" Nov 04 01:35:46.091278 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.091246 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c40ae805-9960-4a82-8ba6-b39b6bc6b458-routing-overlay-volume\") pod \"payload-processing-external-model-5dddfc6875-9n4rm\" (UID: \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\") " pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" Nov 04 01:35:46.091467 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.091315 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkcf4\" (UniqueName: \"kubernetes.io/projected/c40ae805-9960-4a82-8ba6-b39b6bc6b458-kube-api-access-xkcf4\") pod \"payload-processing-external-model-5dddfc6875-9n4rm\" (UID: \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\") " pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" Nov 04 01:35:46.091467 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.091351 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c40ae805-9960-4a82-8ba6-b39b6bc6b458-provider-credentials\") pod \"payload-processing-external-model-5dddfc6875-9n4rm\" (UID: \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\") " pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" Nov 04 01:35:46.091467 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.091395 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c40ae805-9960-4a82-8ba6-b39b6bc6b458-plugins-config-volume\") pod \"payload-processing-external-model-5dddfc6875-9n4rm\" (UID: \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\") " pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" Nov 04 01:35:46.092073 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.092046 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c40ae805-9960-4a82-8ba6-b39b6bc6b458-routing-overlay-volume\") pod \"payload-processing-external-model-5dddfc6875-9n4rm\" (UID: \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\") " pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" Nov 04 01:35:46.092176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.092150 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c40ae805-9960-4a82-8ba6-b39b6bc6b458-plugins-config-volume\") pod \"payload-processing-external-model-5dddfc6875-9n4rm\" (UID: \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\") " pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" Nov 04 01:35:46.093835 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.093809 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c40ae805-9960-4a82-8ba6-b39b6bc6b458-provider-credentials\") pod \"payload-processing-external-model-5dddfc6875-9n4rm\" (UID: \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\") " pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" Nov 04 01:35:46.148979 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.148954 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkcf4\" (UniqueName: \"kubernetes.io/projected/c40ae805-9960-4a82-8ba6-b39b6bc6b458-kube-api-access-xkcf4\") pod \"payload-processing-external-model-5dddfc6875-9n4rm\" (UID: \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\") " pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" Nov 04 01:35:46.192199 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.192154 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" Nov 04 01:35:46.531462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.531350 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g"] Nov 04 01:35:46.535671 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.535646 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g" Nov 04 01:35:46.574456 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.574388 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm"] Nov 04 01:35:46.579764 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:35:46.579730 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40ae805_9960_4a82_8ba6_b39b6bc6b458.slice/crio-646410d5c0f7e3475c1032b4f508cb914a316e89d9a914f90b907883ad1714a8 WatchSource:0}: Error finding container 646410d5c0f7e3475c1032b4f508cb914a316e89d9a914f90b907883ad1714a8: Status 404 returned error can't find the container with id 646410d5c0f7e3475c1032b4f508cb914a316e89d9a914f90b907883ad1714a8 Nov 04 01:35:46.591451 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.590027 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g"] Nov 04 01:35:46.598322 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.598282 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a80ef81-qwx4g\" (UID: \"da967298-5193-4eb8-bbdc-6eaf8ecf5be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g" Nov 04 01:35:46.598489 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.598471 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c649\" (UniqueName: \"kubernetes.io/projected/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-kube-api-access-8c649\") pod \"maas-api-revoke-sub-2a80ef81-qwx4g\" (UID: \"da967298-5193-4eb8-bbdc-6eaf8ecf5be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g" Nov 04 01:35:46.598622 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.598549 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a80ef81-qwx4g\" (UID: \"da967298-5193-4eb8-bbdc-6eaf8ecf5be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g" Nov 04 01:35:46.699481 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.699405 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a80ef81-qwx4g\" (UID: \"da967298-5193-4eb8-bbdc-6eaf8ecf5be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g" Nov 04 01:35:46.700042 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.699531 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a80ef81-qwx4g\" (UID: \"da967298-5193-4eb8-bbdc-6eaf8ecf5be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g" Nov 04 01:35:46.700042 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.699659 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c649\" (UniqueName: \"kubernetes.io/projected/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-kube-api-access-8c649\") pod \"maas-api-revoke-sub-2a80ef81-qwx4g\" (UID: \"da967298-5193-4eb8-bbdc-6eaf8ecf5be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g" Nov 04 01:35:46.700154 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.700083 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a80ef81-qwx4g\" (UID: \"da967298-5193-4eb8-bbdc-6eaf8ecf5be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g" Nov 04 01:35:46.701942 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.701915 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a80ef81-qwx4g\" (UID: \"da967298-5193-4eb8-bbdc-6eaf8ecf5be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g" Nov 04 01:35:46.707699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.707670 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" event={"ID":"c40ae805-9960-4a82-8ba6-b39b6bc6b458","Type":"ContainerStarted","Data":"59be1b727bb9d0b664897d05e1539b08648cc3d290ee4f0c5ef8feb53e9db84c"} Nov 04 01:35:46.707846 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.707721 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" event={"ID":"c40ae805-9960-4a82-8ba6-b39b6bc6b458","Type":"ContainerStarted","Data":"646410d5c0f7e3475c1032b4f508cb914a316e89d9a914f90b907883ad1714a8"} Nov 04 01:35:46.707846 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.707767 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" Nov 04 01:35:46.709394 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.709372 5995 generic.go:356] "Generic (PLEG): container finished" podID="eaafc194-ecb8-46a7-904e-f03823474419" containerID="a10e051d6eb2310000f4fbf625cda1774de124e0ca05f79c5418527a06d8e816" exitCode=0 Nov 04 01:35:46.709510 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.709457 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd" event={"ID":"eaafc194-ecb8-46a7-904e-f03823474419","Type":"ContainerDied","Data":"a10e051d6eb2310000f4fbf625cda1774de124e0ca05f79c5418527a06d8e816"} Nov 04 01:35:46.775558 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.775531 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c649\" (UniqueName: \"kubernetes.io/projected/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-kube-api-access-8c649\") pod \"maas-api-revoke-sub-2a80ef81-qwx4g\" (UID: \"da967298-5193-4eb8-bbdc-6eaf8ecf5be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g" Nov 04 01:35:46.833332 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.833227 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podStartSLOduration=1.833214575 podStartE2EDuration="1.833214575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:46.830819329 +0000 UTC m=+1496.862552734" watchObservedRunningTime="2026-11-04 01:35:46.833214575 +0000 UTC m=+1496.864947965" Nov 04 01:35:46.849396 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:46.849366 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g" Nov 04 01:35:47.226192 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:47.226161 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g"] Nov 04 01:35:47.231781 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:35:47.231750 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda967298_5193_4eb8_bbdc_6eaf8ecf5be3.slice/crio-a7dec9094c9641be4f3109b5a695ba519c1b29d7df56f7d05322bb2dc4e1edc8 WatchSource:0}: Error finding container a7dec9094c9641be4f3109b5a695ba519c1b29d7df56f7d05322bb2dc4e1edc8: Status 404 returned error can't find the container with id a7dec9094c9641be4f3109b5a695ba519c1b29d7df56f7d05322bb2dc4e1edc8 Nov 04 01:35:47.717702 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:47.717657 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g" event={"ID":"da967298-5193-4eb8-bbdc-6eaf8ecf5be3","Type":"ContainerStarted","Data":"66dbbc060ef9726a951a65841e76e9a9c4121e06ffa9eb5f062fdc84d00262e5"} Nov 04 01:35:47.718193 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:47.717723 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g" event={"ID":"da967298-5193-4eb8-bbdc-6eaf8ecf5be3","Type":"ContainerStarted","Data":"a7dec9094c9641be4f3109b5a695ba519c1b29d7df56f7d05322bb2dc4e1edc8"} Nov 04 01:35:47.829659 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:47.829608 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g" podStartSLOduration=1.829593598 podStartE2EDuration="1.829593598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:47.826132363 +0000 UTC m=+1497.857865754" watchObservedRunningTime="2026-11-04 01:35:47.829593598 +0000 UTC m=+1497.861327022" Nov 04 01:35:47.870591 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:47.870566 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd" Nov 04 01:35:47.914317 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:47.914274 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaafc194-ecb8-46a7-904e-f03823474419-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaafc194-ecb8-46a7-904e-f03823474419-maas-api-xks-ca\") pod \"eaafc194-ecb8-46a7-904e-f03823474419\" (UID: \"eaafc194-ecb8-46a7-904e-f03823474419\") " Nov 04 01:35:47.914505 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:47.914487 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaafc194-ecb8-46a7-904e-f03823474419-kube-api-access-9mcrk\" (UniqueName: \"kubernetes.io/projected/eaafc194-ecb8-46a7-904e-f03823474419-kube-api-access-9mcrk\") pod \"eaafc194-ecb8-46a7-904e-f03823474419\" (UID: \"eaafc194-ecb8-46a7-904e-f03823474419\") " Nov 04 01:35:47.914552 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:47.914535 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaafc194-ecb8-46a7-904e-f03823474419-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaafc194-ecb8-46a7-904e-f03823474419-maas-api-openshift-ca\") pod \"eaafc194-ecb8-46a7-904e-f03823474419\" (UID: \"eaafc194-ecb8-46a7-904e-f03823474419\") " Nov 04 01:35:47.914908 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:47.914882 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaafc194-ecb8-46a7-904e-f03823474419-maas-api-openshift-ca" pod "eaafc194-ecb8-46a7-904e-f03823474419" (UID: "eaafc194-ecb8-46a7-904e-f03823474419"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:47.916409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:47.916386 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaafc194-ecb8-46a7-904e-f03823474419-maas-api-xks-ca" pod "eaafc194-ecb8-46a7-904e-f03823474419" (UID: "eaafc194-ecb8-46a7-904e-f03823474419"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:47.916508 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:47.916455 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaafc194-ecb8-46a7-904e-f03823474419-kube-api-access-9mcrk" pod "eaafc194-ecb8-46a7-904e-f03823474419" (UID: "eaafc194-ecb8-46a7-904e-f03823474419"). InnerVolumeSpecName "kube-api-access-9mcrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:48.015858 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:48.015815 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mcrk\" (UniqueName: \"kubernetes.io/projected/eaafc194-ecb8-46a7-904e-f03823474419-kube-api-access-9mcrk\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:48.015858 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:48.015846 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaafc194-ecb8-46a7-904e-f03823474419-maas-api-openshift-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:48.015858 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:48.015856 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaafc194-ecb8-46a7-904e-f03823474419-maas-api-xks-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:48.726613 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:48.726575 5995 generic.go:356] "Generic (PLEG): container finished" podID="da967298-5193-4eb8-bbdc-6eaf8ecf5be3" containerID="66dbbc060ef9726a951a65841e76e9a9c4121e06ffa9eb5f062fdc84d00262e5" exitCode=0 Nov 04 01:35:48.727094 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:48.726655 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g" event={"ID":"da967298-5193-4eb8-bbdc-6eaf8ecf5be3","Type":"ContainerDied","Data":"66dbbc060ef9726a951a65841e76e9a9c4121e06ffa9eb5f062fdc84d00262e5"} Nov 04 01:35:48.728099 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:48.728078 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd" Nov 04 01:35:48.728206 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:48.728077 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd" event={"ID":"eaafc194-ecb8-46a7-904e-f03823474419","Type":"ContainerDied","Data":"419f1cba97bddad9c02fb40954c73b732aa29d75c527022c6ccf1c0ec1e88d67"} Nov 04 01:35:48.728206 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:48.728197 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="419f1cba97bddad9c02fb40954c73b732aa29d75c527022c6ccf1c0ec1e88d67" Nov 04 01:35:49.643455 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:35:49.643405 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:35:49.891650 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:49.891617 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g" Nov 04 01:35:49.938671 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:49.938586 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-kube-api-access-8c649\" (UniqueName: \"kubernetes.io/projected/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-kube-api-access-8c649\") pod \"da967298-5193-4eb8-bbdc-6eaf8ecf5be3\" (UID: \"da967298-5193-4eb8-bbdc-6eaf8ecf5be3\") " Nov 04 01:35:49.938823 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:49.938753 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-maas-api-openshift-ca\") pod \"da967298-5193-4eb8-bbdc-6eaf8ecf5be3\" (UID: \"da967298-5193-4eb8-bbdc-6eaf8ecf5be3\") " Nov 04 01:35:49.938823 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:49.938785 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-maas-api-xks-ca\") pod \"da967298-5193-4eb8-bbdc-6eaf8ecf5be3\" (UID: \"da967298-5193-4eb8-bbdc-6eaf8ecf5be3\") " Nov 04 01:35:49.939104 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:49.939081 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-maas-api-openshift-ca" pod "da967298-5193-4eb8-bbdc-6eaf8ecf5be3" (UID: "da967298-5193-4eb8-bbdc-6eaf8ecf5be3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:49.939232 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:49.939215 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-maas-api-openshift-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:49.940907 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:49.940859 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-maas-api-xks-ca" pod "da967298-5193-4eb8-bbdc-6eaf8ecf5be3" (UID: "da967298-5193-4eb8-bbdc-6eaf8ecf5be3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:49.941077 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:49.940972 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-kube-api-access-8c649" pod "da967298-5193-4eb8-bbdc-6eaf8ecf5be3" (UID: "da967298-5193-4eb8-bbdc-6eaf8ecf5be3"). InnerVolumeSpecName "kube-api-access-8c649". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:50.040317 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:50.040283 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c649\" (UniqueName: \"kubernetes.io/projected/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-kube-api-access-8c649\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:50.040317 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:50.040313 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da967298-5193-4eb8-bbdc-6eaf8ecf5be3-maas-api-xks-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:50.302891 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:50.302863 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f_5600ddae-56c7-4c93-afff-2900207b0819/extract/0.log" Nov 04 01:35:50.304955 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:50.304919 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f_5600ddae-56c7-4c93-afff-2900207b0819/extract/0.log" Nov 04 01:35:50.313309 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:50.313286 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:35:50.314638 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:50.314620 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:35:50.742199 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:50.742169 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g" Nov 04 01:35:50.742199 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:50.742190 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g" event={"ID":"da967298-5193-4eb8-bbdc-6eaf8ecf5be3","Type":"ContainerDied","Data":"a7dec9094c9641be4f3109b5a695ba519c1b29d7df56f7d05322bb2dc4e1edc8"} Nov 04 01:35:50.742458 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:50.742255 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7dec9094c9641be4f3109b5a695ba519c1b29d7df56f7d05322bb2dc4e1edc8" Nov 04 01:35:50.784839 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:50.784779 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn"] Nov 04 01:35:54.013358 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.013318 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx"] Nov 04 01:35:54.014001 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.013983 5995 cpu_manager.go:414] "containerMap: removing container" podUID="da967298-5193-4eb8-bbdc-6eaf8ecf5be3" containerName="revoke-keys" Nov 04 01:35:54.014043 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.014003 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da967298-5193-4eb8-bbdc-6eaf8ecf5be3" containerName="revoke-keys" Nov 04 01:35:54.014043 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.014021 5995 cpu_manager.go:414] "containerMap: removing container" podUID="eaafc194-ecb8-46a7-904e-f03823474419" containerName="revoke-keys" Nov 04 01:35:54.014043 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.014026 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaafc194-ecb8-46a7-904e-f03823474419" containerName="revoke-keys" Nov 04 01:35:54.014147 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.014112 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaafc194-ecb8-46a7-904e-f03823474419" containerName="revoke-keys" Nov 04 01:35:54.014147 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.014127 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="da967298-5193-4eb8-bbdc-6eaf8ecf5be3" containerName="revoke-keys" Nov 04 01:35:54.016949 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.016930 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" Nov 04 01:35:54.067996 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.067961 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-1225f3\"" Nov 04 01:35:54.072847 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.072816 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx"] Nov 04 01:35:54.138452 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.129863 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-1225f3-dockercfg-5cgk8\"" Nov 04 01:35:54.162529 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.161899 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pchr\" (UniqueName: \"kubernetes.io/projected/a5ed3bab-e986-4692-a119-210933b8e3db-kube-api-access-8pchr\") pod \"payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx\" (UID: \"a5ed3bab-e986-4692-a119-210933b8e3db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" Nov 04 01:35:54.162529 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.162003 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ed3bab-e986-4692-a119-210933b8e3db-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx\" (UID: \"a5ed3bab-e986-4692-a119-210933b8e3db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" Nov 04 01:35:54.216920 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.216889 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6"] Nov 04 01:35:54.232784 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.232745 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6" Nov 04 01:35:54.256897 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.256861 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6"] Nov 04 01:35:54.281655 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.268206 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pchr\" (UniqueName: \"kubernetes.io/projected/a5ed3bab-e986-4692-a119-210933b8e3db-kube-api-access-8pchr\") pod \"payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx\" (UID: \"a5ed3bab-e986-4692-a119-210933b8e3db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" Nov 04 01:35:54.281655 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.268316 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bd339e-06fd-46be-b1c7-8fa2792d2a7e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6\" (UID: \"d2bd339e-06fd-46be-b1c7-8fa2792d2a7e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6" Nov 04 01:35:54.281655 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.268375 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ed3bab-e986-4692-a119-210933b8e3db-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx\" (UID: \"a5ed3bab-e986-4692-a119-210933b8e3db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" Nov 04 01:35:54.281655 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.279600 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9257l\" (UniqueName: \"kubernetes.io/projected/d2bd339e-06fd-46be-b1c7-8fa2792d2a7e-kube-api-access-9257l\") pod \"payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6\" (UID: \"d2bd339e-06fd-46be-b1c7-8fa2792d2a7e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6" Nov 04 01:35:54.281655 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.280634 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ed3bab-e986-4692-a119-210933b8e3db-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx\" (UID: \"a5ed3bab-e986-4692-a119-210933b8e3db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" Nov 04 01:35:54.338609 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.338572 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pchr\" (UniqueName: \"kubernetes.io/projected/a5ed3bab-e986-4692-a119-210933b8e3db-kube-api-access-8pchr\") pod \"payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx\" (UID: \"a5ed3bab-e986-4692-a119-210933b8e3db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" Nov 04 01:35:54.380961 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.380926 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9257l\" (UniqueName: \"kubernetes.io/projected/d2bd339e-06fd-46be-b1c7-8fa2792d2a7e-kube-api-access-9257l\") pod \"payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6\" (UID: \"d2bd339e-06fd-46be-b1c7-8fa2792d2a7e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6" Nov 04 01:35:54.381151 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.381131 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bd339e-06fd-46be-b1c7-8fa2792d2a7e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6\" (UID: \"d2bd339e-06fd-46be-b1c7-8fa2792d2a7e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6" Nov 04 01:35:54.381943 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.381914 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bd339e-06fd-46be-b1c7-8fa2792d2a7e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6\" (UID: \"d2bd339e-06fd-46be-b1c7-8fa2792d2a7e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6" Nov 04 01:35:54.438204 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.438171 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9257l\" (UniqueName: \"kubernetes.io/projected/d2bd339e-06fd-46be-b1c7-8fa2792d2a7e-kube-api-access-9257l\") pod \"payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6\" (UID: \"d2bd339e-06fd-46be-b1c7-8fa2792d2a7e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6" Nov 04 01:35:54.592335 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.592223 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6" Nov 04 01:35:54.637196 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.637163 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" Nov 04 01:35:54.970330 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:54.970292 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6"] Nov 04 01:35:54.974638 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:35:54.974597 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2bd339e_06fd_46be_b1c7_8fa2792d2a7e.slice/crio-e8e442ddca31409ccaac9a071b54405f45cab9afa9a88c4cf20bfb52e6a8dfad WatchSource:0}: Error finding container e8e442ddca31409ccaac9a071b54405f45cab9afa9a88c4cf20bfb52e6a8dfad: Status 404 returned error can't find the container with id e8e442ddca31409ccaac9a071b54405f45cab9afa9a88c4cf20bfb52e6a8dfad Nov 04 01:35:55.097720 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:55.094378 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx"] Nov 04 01:35:55.308818 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:55.308624 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:35:55.835205 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:55.835166 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6" event={"ID":"d2bd339e-06fd-46be-b1c7-8fa2792d2a7e","Type":"ContainerStarted","Data":"f129b0e6584bb39462756835cd2058db00a4d638fec76d5b0eeac638f316b849"} Nov 04 01:35:55.835370 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:55.835231 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6" Nov 04 01:35:55.835370 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:55.835256 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6" event={"ID":"d2bd339e-06fd-46be-b1c7-8fa2792d2a7e","Type":"ContainerStarted","Data":"e8e442ddca31409ccaac9a071b54405f45cab9afa9a88c4cf20bfb52e6a8dfad"} Nov 04 01:35:55.836945 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:55.836920 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" event={"ID":"a5ed3bab-e986-4692-a119-210933b8e3db","Type":"ContainerStarted","Data":"a5afce8b65604fb1711afe3440b28647a9d348558d75a3fb92ccde6de116c7c3"} Nov 04 01:35:55.837066 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:55.836964 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" event={"ID":"a5ed3bab-e986-4692-a119-210933b8e3db","Type":"ContainerStarted","Data":"971596ddd6472d3fd135bb886e987659c9fa3f05f1ec283da4693a7407b2abef"} Nov 04 01:35:55.837066 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:55.837014 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" Nov 04 01:35:55.946279 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:55.946226 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6" podStartSLOduration=1.946213647 podStartE2EDuration="1.946213647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:55.943954918 +0000 UTC m=+1505.975688308" watchObservedRunningTime="2026-11-04 01:35:55.946213647 +0000 UTC m=+1505.977947036" Nov 04 01:35:56.070048 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:56.060943 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" podStartSLOduration=3.060923083 podStartE2EDuration="3.060923083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:56.057288468 +0000 UTC m=+1506.089021861" watchObservedRunningTime="2026-11-04 01:35:56.060923083 +0000 UTC m=+1506.092656475" Nov 04 01:35:57.720326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:57.720289 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:35:58.053166 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.053053 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx"] Nov 04 01:35:58.057501 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.057481 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.109439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.107266 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4"] Nov 04 01:35:58.115437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.113205 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.224440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.223809 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.224440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.223975 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.224440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.224026 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.224440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.224103 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370be073-d29b-47c0-8091-bee82e0b9dbb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.224440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.224130 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.224440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.224195 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.224440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.224262 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.244442 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.225535 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.244442 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.225617 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d392036c-7164-4328-9d05-a52e2e9eec82-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.244442 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.225692 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.244442 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.225797 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d392036c-7164-4328-9d05-a52e2e9eec82-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.244442 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.225828 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.244442 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.225926 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.244442 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.225965 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvdt6\" (UniqueName: \"kubernetes.io/projected/d392036c-7164-4328-9d05-a52e2e9eec82-kube-api-access-zvdt6\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.244442 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.226045 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d392036c-7164-4328-9d05-a52e2e9eec82-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.244442 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.226079 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwfz2\" (UniqueName: \"kubernetes.io/projected/370be073-d29b-47c0-8091-bee82e0b9dbb-kube-api-access-rwfz2\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.244442 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.226112 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.244442 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.226180 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.246554 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.246455 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4"] Nov 04 01:35:58.260609 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.260142 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx"] Nov 04 01:35:58.334379 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.334279 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.336486 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.336449 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d392036c-7164-4328-9d05-a52e2e9eec82-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.336662 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.336531 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.336662 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.336569 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d392036c-7164-4328-9d05-a52e2e9eec82-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.336662 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.336614 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.336841 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.336680 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.336841 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.336722 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvdt6\" (UniqueName: \"kubernetes.io/projected/d392036c-7164-4328-9d05-a52e2e9eec82-kube-api-access-zvdt6\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.336841 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.336783 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d392036c-7164-4328-9d05-a52e2e9eec82-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.336841 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.336808 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwfz2\" (UniqueName: \"kubernetes.io/projected/370be073-d29b-47c0-8091-bee82e0b9dbb-kube-api-access-rwfz2\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.337059 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.336856 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.337059 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.336985 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.337059 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.337032 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.337213 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.337122 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.337213 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.337171 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.337318 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.337201 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370be073-d29b-47c0-8091-bee82e0b9dbb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.337318 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.337240 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.337318 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.337274 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.337515 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.337323 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.339108 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.338236 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.339108 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.338532 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.339108 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.338627 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.339108 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.338715 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.339108 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.338862 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.339404 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.339136 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370be073-d29b-47c0-8091-bee82e0b9dbb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.339404 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.339382 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.339532 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.339456 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.339712 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.339690 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.339807 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.339784 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d392036c-7164-4328-9d05-a52e2e9eec82-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.341429 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.341391 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.342051 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.342035 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d392036c-7164-4328-9d05-a52e2e9eec82-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.342852 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.342829 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.343540 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.343522 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.403689 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.403552 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.403923 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.403898 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d392036c-7164-4328-9d05-a52e2e9eec82-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.409095 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.406146 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwfz2\" (UniqueName: \"kubernetes.io/projected/370be073-d29b-47c0-8091-bee82e0b9dbb-kube-api-access-rwfz2\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b7kv4\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.409095 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.409047 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvdt6\" (UniqueName: \"kubernetes.io/projected/d392036c-7164-4328-9d05-a52e2e9eec82-kube-api-access-zvdt6\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-449zx\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.441446 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.441403 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:35:58.707060 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.707016 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:35:58.950158 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.950122 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4"] Nov 04 01:35:58.955072 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:35:58.955033 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370be073_d29b_47c0_8091_bee82e0b9dbb.slice/crio-a6f7503e489a3acd3169608f2d16dc1023257cf3b20e832bec1d5b3e7bcc7b7a WatchSource:0}: Error finding container a6f7503e489a3acd3169608f2d16dc1023257cf3b20e832bec1d5b3e7bcc7b7a: Status 404 returned error can't find the container with id a6f7503e489a3acd3169608f2d16dc1023257cf3b20e832bec1d5b3e7bcc7b7a Nov 04 01:35:58.958510 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.958465 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:35:58.958604 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.958562 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:35:58.958670 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:58.958605 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:35:59.131790 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:59.131752 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx"] Nov 04 01:35:59.138597 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:35:59.138558 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd392036c_7164_4328_9d05_a52e2e9eec82.slice/crio-00195f85070adf87268bf50452362a6a6592c2a2bfcab7f61a43e254e83d4cd8 WatchSource:0}: Error finding container 00195f85070adf87268bf50452362a6a6592c2a2bfcab7f61a43e254e83d4cd8: Status 404 returned error can't find the container with id 00195f85070adf87268bf50452362a6a6592c2a2bfcab7f61a43e254e83d4cd8 Nov 04 01:35:59.141523 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:59.141483 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:35:59.141623 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:59.141611 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:35:59.141681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:59.141649 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:35:59.645240 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:59.645147 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch"] Nov 04 01:35:59.946106 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:59.945015 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" event={"ID":"d392036c-7164-4328-9d05-a52e2e9eec82","Type":"ContainerStarted","Data":"b614eeedcb1f460d17dd92195d4796068f833d0a43e818cfb3f40ab5ee7d759e"} Nov 04 01:35:59.946106 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:59.945080 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" event={"ID":"d392036c-7164-4328-9d05-a52e2e9eec82","Type":"ContainerStarted","Data":"00195f85070adf87268bf50452362a6a6592c2a2bfcab7f61a43e254e83d4cd8"} Nov 04 01:35:59.948606 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:59.948571 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" event={"ID":"370be073-d29b-47c0-8091-bee82e0b9dbb","Type":"ContainerStarted","Data":"594cacaefc01aff34eb527ae37a8b2ff04d85495a5c1a02556a02946d8c09061"} Nov 04 01:35:59.948846 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:35:59.948830 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" event={"ID":"370be073-d29b-47c0-8091-bee82e0b9dbb","Type":"ContainerStarted","Data":"a6f7503e489a3acd3169608f2d16dc1023257cf3b20e832bec1d5b3e7bcc7b7a"} Nov 04 01:36:00.064994 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:00.064930 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" podStartSLOduration=2.06491143 podStartE2EDuration="2.06491143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:36:00.063491605 +0000 UTC m=+1510.095225005" watchObservedRunningTime="2026-11-04 01:36:00.06491143 +0000 UTC m=+1510.096644820" Nov 04 01:36:00.179567 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:00.179515 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" podStartSLOduration=2.179500229 podStartE2EDuration="2.179500229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:36:00.177056366 +0000 UTC m=+1510.208789756" watchObservedRunningTime="2026-11-04 01:36:00.179500229 +0000 UTC m=+1510.211233619" Nov 04 01:36:00.442530 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:00.441761 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:36:00.463439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:00.453691 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:36:00.463439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:00.453754 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" podUID="370be073-d29b-47c0-8091-bee82e0b9dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 04 01:36:00.711495 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:00.710856 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:36:00.711495 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:00.711134 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-449zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 04 01:36:00.711495 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:00.711183 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" podUID="d392036c-7164-4328-9d05-a52e2e9eec82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 04 01:36:01.442249 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:01.442208 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:36:01.442688 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:01.442266 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" podUID="370be073-d29b-47c0-8091-bee82e0b9dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 04 01:36:01.716441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:01.715851 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-449zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 04 01:36:01.716441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:01.715932 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" podUID="d392036c-7164-4328-9d05-a52e2e9eec82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 04 01:36:02.442493 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.442458 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:36:02.442889 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.442512 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" podUID="370be073-d29b-47c0-8091-bee82e0b9dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 04 01:36:02.590440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.587401 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9"] Nov 04 01:36:02.611440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.598072 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9" Nov 04 01:36:02.642965 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.642931 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9"] Nov 04 01:36:02.651168 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.651131 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:36:02.653471 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.652971 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd33b9f-6727-4178-b340-6845fe234a1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00177864-jg8m9\" (UID: \"bdd33b9f-6727-4178-b340-6845fe234a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9" Nov 04 01:36:02.653471 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.653024 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd33b9f-6727-4178-b340-6845fe234a1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00177864-jg8m9\" (UID: \"bdd33b9f-6727-4178-b340-6845fe234a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9" Nov 04 01:36:02.653471 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.653130 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vdd7\" (UniqueName: \"kubernetes.io/projected/bdd33b9f-6727-4178-b340-6845fe234a1d-kube-api-access-4vdd7\") pod \"maas-api-revoke-sub-00177864-jg8m9\" (UID: \"bdd33b9f-6727-4178-b340-6845fe234a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9" Nov 04 01:36:02.696508 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.684779 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:36:02.696508 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.684888 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w799x\"" Nov 04 01:36:02.696508 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.684961 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:36:02.733194 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.722538 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-449zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 04 01:36:02.733194 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.722640 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" podUID="d392036c-7164-4328-9d05-a52e2e9eec82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 04 01:36:02.754739 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.754706 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd33b9f-6727-4178-b340-6845fe234a1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00177864-jg8m9\" (UID: \"bdd33b9f-6727-4178-b340-6845fe234a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9" Nov 04 01:36:02.754921 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.754769 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd33b9f-6727-4178-b340-6845fe234a1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00177864-jg8m9\" (UID: \"bdd33b9f-6727-4178-b340-6845fe234a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9" Nov 04 01:36:02.754921 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.754892 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vdd7\" (UniqueName: \"kubernetes.io/projected/bdd33b9f-6727-4178-b340-6845fe234a1d-kube-api-access-4vdd7\") pod \"maas-api-revoke-sub-00177864-jg8m9\" (UID: \"bdd33b9f-6727-4178-b340-6845fe234a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9" Nov 04 01:36:02.756027 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.756002 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd33b9f-6727-4178-b340-6845fe234a1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00177864-jg8m9\" (UID: \"bdd33b9f-6727-4178-b340-6845fe234a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9" Nov 04 01:36:02.757473 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.757453 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd33b9f-6727-4178-b340-6845fe234a1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00177864-jg8m9\" (UID: \"bdd33b9f-6727-4178-b340-6845fe234a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9" Nov 04 01:36:02.825437 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.824704 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vdd7\" (UniqueName: \"kubernetes.io/projected/bdd33b9f-6727-4178-b340-6845fe234a1d-kube-api-access-4vdd7\") pod \"maas-api-revoke-sub-00177864-jg8m9\" (UID: \"bdd33b9f-6727-4178-b340-6845fe234a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9" Nov 04 01:36:02.940001 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:02.939967 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9" Nov 04 01:36:03.425493 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:03.425181 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9"] Nov 04 01:36:03.442278 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:03.442246 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:36:03.442395 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:03.442305 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" podUID="370be073-d29b-47c0-8091-bee82e0b9dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 04 01:36:03.712453 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:03.711867 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-449zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 04 01:36:03.712453 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:03.711930 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" podUID="d392036c-7164-4328-9d05-a52e2e9eec82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 04 01:36:04.036172 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:04.036139 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9" event={"ID":"bdd33b9f-6727-4178-b340-6845fe234a1d","Type":"ContainerStarted","Data":"bfdd7bb525fac34d93c51e5a580acb5be7ea260281cb79b132c5c4ed63c6168e"} Nov 04 01:36:04.147205 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:04.147131 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9" podStartSLOduration=2.147117949 podStartE2EDuration="2.147117949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:36:04.145141575 +0000 UTC m=+1514.176874975" watchObservedRunningTime="2026-11-04 01:36:04.147117949 +0000 UTC m=+1514.178851358" Nov 04 01:36:04.442452 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:04.442347 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:36:04.442452 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:04.442403 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" podUID="370be073-d29b-47c0-8091-bee82e0b9dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 04 01:36:04.646227 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:04.646178 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" podUID="3443e17f-4370-4c7e-8d10-8a79d491dca4" containerName="payload-processing" containerID="cri-o://145070b458cb9dd8be2cb021cebcc84fb721896fce134d316f8b50a4b36c05d4" gracePeriod=25 Nov 04 01:36:04.713808 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:04.711128 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-449zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 04 01:36:04.713808 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:04.711185 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" podUID="d392036c-7164-4328-9d05-a52e2e9eec82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 04 01:36:04.828451 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:04.826169 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq"] Nov 04 01:36:04.952572 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:04.952544 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx"] Nov 04 01:36:05.047045 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.046953 5995 generic.go:356] "Generic (PLEG): container finished" podID="bdd33b9f-6727-4178-b340-6845fe234a1d" containerID="5a7fa114e698b5831affaf6fa89f71cbe4b8223fd7da24312d1cd9f4fda6462f" exitCode=0 Nov 04 01:36:05.047227 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.047013 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9" event={"ID":"bdd33b9f-6727-4178-b340-6845fe234a1d","Type":"ContainerDied","Data":"5a7fa114e698b5831affaf6fa89f71cbe4b8223fd7da24312d1cd9f4fda6462f"} Nov 04 01:36:05.061213 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.061176 5995 generic.go:356] "Generic (PLEG): container finished" podID="3443e17f-4370-4c7e-8d10-8a79d491dca4" containerID="145070b458cb9dd8be2cb021cebcc84fb721896fce134d316f8b50a4b36c05d4" exitCode=0 Nov 04 01:36:05.061365 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.061243 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" event={"ID":"3443e17f-4370-4c7e-8d10-8a79d491dca4","Type":"ContainerDied","Data":"145070b458cb9dd8be2cb021cebcc84fb721896fce134d316f8b50a4b36c05d4"} Nov 04 01:36:05.244686 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.244648 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" Nov 04 01:36:05.440101 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.440001 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3443e17f-4370-4c7e-8d10-8a79d491dca4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3443e17f-4370-4c7e-8d10-8a79d491dca4-routing-overlay-volume\") pod \"3443e17f-4370-4c7e-8d10-8a79d491dca4\" (UID: \"3443e17f-4370-4c7e-8d10-8a79d491dca4\") " Nov 04 01:36:05.440433 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.440391 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3443e17f-4370-4c7e-8d10-8a79d491dca4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3443e17f-4370-4c7e-8d10-8a79d491dca4-plugins-config-volume\") pod \"3443e17f-4370-4c7e-8d10-8a79d491dca4\" (UID: \"3443e17f-4370-4c7e-8d10-8a79d491dca4\") " Nov 04 01:36:05.440657 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.440636 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3443e17f-4370-4c7e-8d10-8a79d491dca4-kube-api-access-qm4m5\" (UniqueName: \"kubernetes.io/projected/3443e17f-4370-4c7e-8d10-8a79d491dca4-kube-api-access-qm4m5\") pod \"3443e17f-4370-4c7e-8d10-8a79d491dca4\" (UID: \"3443e17f-4370-4c7e-8d10-8a79d491dca4\") " Nov 04 01:36:05.440812 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.440797 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3443e17f-4370-4c7e-8d10-8a79d491dca4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3443e17f-4370-4c7e-8d10-8a79d491dca4-provider-credentials\") pod \"3443e17f-4370-4c7e-8d10-8a79d491dca4\" (UID: \"3443e17f-4370-4c7e-8d10-8a79d491dca4\") " Nov 04 01:36:05.466115 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.466075 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3443e17f-4370-4c7e-8d10-8a79d491dca4-plugins-config-volume" pod "3443e17f-4370-4c7e-8d10-8a79d491dca4" (UID: "3443e17f-4370-4c7e-8d10-8a79d491dca4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:05.466675 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.466654 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3443e17f-4370-4c7e-8d10-8a79d491dca4-routing-overlay-volume" pod "3443e17f-4370-4c7e-8d10-8a79d491dca4" (UID: "3443e17f-4370-4c7e-8d10-8a79d491dca4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:05.502722 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.470778 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:36:05.531479 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.506517 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" podUID="370be073-d29b-47c0-8091-bee82e0b9dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 04 01:36:05.556438 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.553978 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3443e17f-4370-4c7e-8d10-8a79d491dca4-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:05.556438 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.554016 5995 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3443e17f-4370-4c7e-8d10-8a79d491dca4-routing-overlay-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:05.595901 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.595265 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3443e17f-4370-4c7e-8d10-8a79d491dca4-provider-credentials" pod "3443e17f-4370-4c7e-8d10-8a79d491dca4" (UID: "3443e17f-4370-4c7e-8d10-8a79d491dca4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:05.633470 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.610578 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3443e17f-4370-4c7e-8d10-8a79d491dca4-kube-api-access-qm4m5" pod "3443e17f-4370-4c7e-8d10-8a79d491dca4" (UID: "3443e17f-4370-4c7e-8d10-8a79d491dca4"). InnerVolumeSpecName "kube-api-access-qm4m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:05.684376 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.684294 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm4m5\" (UniqueName: \"kubernetes.io/projected/3443e17f-4370-4c7e-8d10-8a79d491dca4-kube-api-access-qm4m5\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:05.684376 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.684331 5995 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3443e17f-4370-4c7e-8d10-8a79d491dca4-provider-credentials\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:05.742494 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.717218 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-449zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 04 01:36:05.742494 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.717289 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" podUID="d392036c-7164-4328-9d05-a52e2e9eec82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 04 01:36:05.810710 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:05.810027 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" podUID="4bbe9a41-a5fc-4ee3-98e2-18be1232cffb" containerName="main" containerID="cri-o://68d49da2a401828d49678a3b1cd7a2e10dbbbbc9319c567b6776033c7ea10aaa" gracePeriod=45 Nov 04 01:36:06.019010 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.018012 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" podUID="4bbe9a41-a5fc-4ee3-98e2-18be1232cffb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": read tcp 10.132.0.2:48602->10.132.0.66:8000: read: connection reset by peer" Nov 04 01:36:06.076396 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.076357 5995 generic.go:356] "Generic (PLEG): container finished" podID="4bbe9a41-a5fc-4ee3-98e2-18be1232cffb" containerID="68d49da2a401828d49678a3b1cd7a2e10dbbbbc9319c567b6776033c7ea10aaa" exitCode=0 Nov 04 01:36:06.076621 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.076548 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" event={"ID":"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb","Type":"ContainerDied","Data":"68d49da2a401828d49678a3b1cd7a2e10dbbbbc9319c567b6776033c7ea10aaa"} Nov 04 01:36:06.082326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.078782 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" Nov 04 01:36:06.082326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.081493 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch" event={"ID":"3443e17f-4370-4c7e-8d10-8a79d491dca4","Type":"ContainerDied","Data":"9f5a34038446e1ba052f401fcf89f65cc133c51d310de342e15f66c458da1ba6"} Nov 04 01:36:06.082326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.081552 5995 scope.go:122] "RemoveContainer" containerID="145070b458cb9dd8be2cb021cebcc84fb721896fce134d316f8b50a4b36c05d4" Nov 04 01:36:06.247241 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.247207 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch"] Nov 04 01:36:06.259452 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.252738 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-cae0ab/payload-processing-external-model-e2e-extmod-b-cae0ab-856dlf7ch"] Nov 04 01:36:06.437629 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.434594 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9" Nov 04 01:36:06.443249 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.442855 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:36:06.443249 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.442918 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" podUID="370be073-d29b-47c0-8091-bee82e0b9dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 04 01:36:06.524982 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.524937 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd33b9f-6727-4178-b340-6845fe234a1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd33b9f-6727-4178-b340-6845fe234a1d-maas-api-openshift-ca\") pod \"bdd33b9f-6727-4178-b340-6845fe234a1d\" (UID: \"bdd33b9f-6727-4178-b340-6845fe234a1d\") " Nov 04 01:36:06.567436 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.567336 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd33b9f-6727-4178-b340-6845fe234a1d-kube-api-access-4vdd7\" (UniqueName: \"kubernetes.io/projected/bdd33b9f-6727-4178-b340-6845fe234a1d-kube-api-access-4vdd7\") pod \"bdd33b9f-6727-4178-b340-6845fe234a1d\" (UID: \"bdd33b9f-6727-4178-b340-6845fe234a1d\") " Nov 04 01:36:06.567711 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.567686 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdd33b9f-6727-4178-b340-6845fe234a1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd33b9f-6727-4178-b340-6845fe234a1d-maas-api-xks-ca\") pod \"bdd33b9f-6727-4178-b340-6845fe234a1d\" (UID: \"bdd33b9f-6727-4178-b340-6845fe234a1d\") " Nov 04 01:36:06.567834 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.545093 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd33b9f-6727-4178-b340-6845fe234a1d-maas-api-openshift-ca" pod "bdd33b9f-6727-4178-b340-6845fe234a1d" (UID: "bdd33b9f-6727-4178-b340-6845fe234a1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:06.571618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.571598 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd33b9f-6727-4178-b340-6845fe234a1d-maas-api-openshift-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.635438 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.629705 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd33b9f-6727-4178-b340-6845fe234a1d-kube-api-access-4vdd7" pod "bdd33b9f-6727-4178-b340-6845fe234a1d" (UID: "bdd33b9f-6727-4178-b340-6845fe234a1d"). InnerVolumeSpecName "kube-api-access-4vdd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:06.636265 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.629869 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdd33b9f-6727-4178-b340-6845fe234a1d-maas-api-xks-ca" pod "bdd33b9f-6727-4178-b340-6845fe234a1d" (UID: "bdd33b9f-6727-4178-b340-6845fe234a1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:36:06.642685 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.641989 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" podUID="4bbe9a41-a5fc-4ee3-98e2-18be1232cffb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": dial tcp 10.132.0.66:8000: connect: connection refused" Nov 04 01:36:06.642685 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.642123 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:36:06.675546 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.675481 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vdd7\" (UniqueName: \"kubernetes.io/projected/bdd33b9f-6727-4178-b340-6845fe234a1d-kube-api-access-4vdd7\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.675546 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.675517 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd33b9f-6727-4178-b340-6845fe234a1d-maas-api-xks-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.710567 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.710526 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-449zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 04 01:36:06.710969 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.710937 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" podUID="d392036c-7164-4328-9d05-a52e2e9eec82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 04 01:36:06.853981 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.853942 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" Nov 04 01:36:06.855065 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.855034 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6" Nov 04 01:36:06.923465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.923406 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:36:06.979607 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.979569 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-model-cache\") pod \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " Nov 04 01:36:06.979798 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.979671 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-kube-api-access-pnv8x\" (UniqueName: \"kubernetes.io/projected/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-kube-api-access-pnv8x\") pod \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " Nov 04 01:36:06.979798 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.979709 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-tmp-dir\") pod \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " Nov 04 01:36:06.979924 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.979901 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-tls-certs\" (UniqueName: \"kubernetes.io/secret/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-tls-certs\") pod \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " Nov 04 01:36:06.979924 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.979894 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-model-cache" pod "4bbe9a41-a5fc-4ee3-98e2-18be1232cffb" (UID: "4bbe9a41-a5fc-4ee3-98e2-18be1232cffb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.980037 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.979962 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-home\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-home\") pod \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " Nov 04 01:36:06.980037 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.979996 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-tmp-dir" pod "4bbe9a41-a5fc-4ee3-98e2-18be1232cffb" (UID: "4bbe9a41-a5fc-4ee3-98e2-18be1232cffb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.980164 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.980085 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-dshm\") pod \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\" (UID: \"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb\") " Nov 04 01:36:06.980309 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.980267 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-home" pod "4bbe9a41-a5fc-4ee3-98e2-18be1232cffb" (UID: "4bbe9a41-a5fc-4ee3-98e2-18be1232cffb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.981379 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.980625 5995 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-model-cache\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.981379 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.980650 5995 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-tmp-dir\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.981379 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.980662 5995 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-home\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.982758 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.982686 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-kube-api-access-pnv8x" pod "4bbe9a41-a5fc-4ee3-98e2-18be1232cffb" (UID: "4bbe9a41-a5fc-4ee3-98e2-18be1232cffb"). InnerVolumeSpecName "kube-api-access-pnv8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:06.982758 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.982721 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-dshm" pod "4bbe9a41-a5fc-4ee3-98e2-18be1232cffb" (UID: "4bbe9a41-a5fc-4ee3-98e2-18be1232cffb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.982912 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:06.982762 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-tls-certs" pod "4bbe9a41-a5fc-4ee3-98e2-18be1232cffb" (UID: "4bbe9a41-a5fc-4ee3-98e2-18be1232cffb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:36:07.090097 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:07.089941 5995 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-dshm\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:07.090097 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:07.089980 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnv8x\" (UniqueName: \"kubernetes.io/projected/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-kube-api-access-pnv8x\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:07.090097 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:07.089996 5995 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb-tls-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:07.109456 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:07.108359 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" Nov 04 01:36:07.109456 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:07.108371 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn" event={"ID":"4bbe9a41-a5fc-4ee3-98e2-18be1232cffb","Type":"ContainerDied","Data":"4892fedc615e4057fb4e1de0ab085efc34dcf7681d97bfd92fc8acdfdbe6338d"} Nov 04 01:36:07.109456 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:07.108455 5995 scope.go:122] "RemoveContainer" containerID="68d49da2a401828d49678a3b1cd7a2e10dbbbbc9319c567b6776033c7ea10aaa" Nov 04 01:36:07.119353 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:07.119272 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9" event={"ID":"bdd33b9f-6727-4178-b340-6845fe234a1d","Type":"ContainerDied","Data":"bfdd7bb525fac34d93c51e5a580acb5be7ea260281cb79b132c5c4ed63c6168e"} Nov 04 01:36:07.121722 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:07.121686 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfdd7bb525fac34d93c51e5a580acb5be7ea260281cb79b132c5c4ed63c6168e" Nov 04 01:36:07.141033 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:07.129949 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9" Nov 04 01:36:07.362689 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:07.362582 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn"] Nov 04 01:36:07.368702 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:07.368669 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1baf54/distinct-w1-1baf54-kserve-6f746d7785-rflcn"] Nov 04 01:36:07.442802 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:07.442761 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:36:07.442979 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:07.442836 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" podUID="370be073-d29b-47c0-8091-bee82e0b9dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 04 01:36:07.708607 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:07.708563 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-449zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 04 01:36:07.708811 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:07.708641 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" podUID="d392036c-7164-4328-9d05-a52e2e9eec82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 04 01:36:07.719999 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:07.719962 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:36:08.131871 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:08.131782 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3443e17f-4370-4c7e-8d10-8a79d491dca4" path="/var/lib/kubelet/pods/3443e17f-4370-4c7e-8d10-8a79d491dca4/volumes" Nov 04 01:36:08.132505 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:08.132484 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bbe9a41-a5fc-4ee3-98e2-18be1232cffb" path="/var/lib/kubelet/pods/4bbe9a41-a5fc-4ee3-98e2-18be1232cffb/volumes" Nov 04 01:36:09.443464 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:09.443363 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:36:09.443464 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:09.443447 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" podUID="370be073-d29b-47c0-8091-bee82e0b9dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:36:09.474927 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:09.474894 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:36:09.476140 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:09.476116 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:36:09.477035 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:09.477015 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:36:09.479095 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:09.479066 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:36:09.479624 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:09.479606 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:36:09.480526 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:09.480508 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:36:09.827543 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:09.827394 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" podUID="2ecb79d2-6f5f-489f-b80d-de78155993b9" containerName="payload-processing" containerID="cri-o://505324c77f260825daf96fe9f803b0c07a690177d6de53e49e2edd31c737280e" gracePeriod=25 Nov 04 01:36:09.953517 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:09.953474 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" podUID="290fa234-8b22-4932-a683-a782790c388d" containerName="payload-pre-processing" containerID="cri-o://8f118cf833e12f434de0d4e1314c1e07a803dd3c6e6dce59eeafed00ce9581fd" gracePeriod=25 Nov 04 01:36:11.855286 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:11.855253 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" podUID="290fa234-8b22-4932-a683-a782790c388d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:36:12.860599 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:12.860561 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" podUID="2ecb79d2-6f5f-489f-b80d-de78155993b9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:36:17.719714 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:17.719674 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:36:19.643534 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:19.643492 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:36:21.854978 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:21.854940 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" podUID="290fa234-8b22-4932-a683-a782790c388d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:36:22.146533 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:22.146438 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bec17d7cdfd4: secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:22.146533 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:22.146472 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz: secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:22.146746 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:22.146542 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials podName:7650b8ae-725e-470e-8960-5053aa977ea0 nodeName:}" failed. No retries permitted until 2026-11-04 01:36:22.64652632 +0000 UTC m=+1532.678259704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials") pod "payload-processing-external-model-6fddb87f65-hxqhz" (UID: "7650b8ae-725e-470e-8960-5053aa977ea0") : secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:22.651406 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:22.651370 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bec17d7cdfd4: secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:22.651406 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:22.651403 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz: secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:22.651619 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:22.651488 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials podName:7650b8ae-725e-470e-8960-5053aa977ea0 nodeName:}" failed. No retries permitted until 2026-11-04 01:36:23.651472287 +0000 UTC m=+1533.683205678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials") pod "payload-processing-external-model-6fddb87f65-hxqhz" (UID: "7650b8ae-725e-470e-8960-5053aa977ea0") : secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:22.860805 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:22.860769 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" podUID="2ecb79d2-6f5f-489f-b80d-de78155993b9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:36:23.664073 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:23.664029 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bec17d7cdfd4: secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:23.664073 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:23.664064 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz: secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:23.664302 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:23.664121 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials podName:7650b8ae-725e-470e-8960-5053aa977ea0 nodeName:}" failed. No retries permitted until 2026-11-04 01:36:25.664105371 +0000 UTC m=+1535.695838739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials") pod "payload-processing-external-model-6fddb87f65-hxqhz" (UID: "7650b8ae-725e-470e-8960-5053aa977ea0") : secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:25.694537 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:25.694500 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bec17d7cdfd4: secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:25.694537 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:25.694541 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz: secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:25.694997 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:25.694606 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials podName:7650b8ae-725e-470e-8960-5053aa977ea0 nodeName:}" failed. No retries permitted until 2026-11-04 01:36:29.694591172 +0000 UTC m=+1539.726324545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials") pod "payload-processing-external-model-6fddb87f65-hxqhz" (UID: "7650b8ae-725e-470e-8960-5053aa977ea0") : secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:26.673884 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:26.673843 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn"] Nov 04 01:36:26.819438 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:26.818531 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt"] Nov 04 01:36:27.719742 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:27.719706 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:36:29.719540 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:29.719504 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bec17d7cdfd4: secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:29.719540 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:29.719543 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz: secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:29.720059 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:29.719621 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials podName:7650b8ae-725e-470e-8960-5053aa977ea0 nodeName:}" failed. No retries permitted until 2026-11-04 01:36:37.71959961 +0000 UTC m=+1547.751332998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials") pod "payload-processing-external-model-6fddb87f65-hxqhz" (UID: "7650b8ae-725e-470e-8960-5053aa977ea0") : secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:30.101985 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.101959 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" Nov 04 01:36:30.216923 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.216898 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" Nov 04 01:36:30.226552 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.226485 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ecb79d2-6f5f-489f-b80d-de78155993b9-kube-api-access-4vshq\" (UniqueName: \"kubernetes.io/projected/2ecb79d2-6f5f-489f-b80d-de78155993b9-kube-api-access-4vshq\") pod \"2ecb79d2-6f5f-489f-b80d-de78155993b9\" (UID: \"2ecb79d2-6f5f-489f-b80d-de78155993b9\") " Nov 04 01:36:30.226679 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.226605 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ecb79d2-6f5f-489f-b80d-de78155993b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb79d2-6f5f-489f-b80d-de78155993b9-plugins-config-volume\") pod \"2ecb79d2-6f5f-489f-b80d-de78155993b9\" (UID: \"2ecb79d2-6f5f-489f-b80d-de78155993b9\") " Nov 04 01:36:30.226977 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.226948 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ecb79d2-6f5f-489f-b80d-de78155993b9-plugins-config-volume" pod "2ecb79d2-6f5f-489f-b80d-de78155993b9" (UID: "2ecb79d2-6f5f-489f-b80d-de78155993b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:30.227149 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.227132 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb79d2-6f5f-489f-b80d-de78155993b9-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:30.228623 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.228604 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecb79d2-6f5f-489f-b80d-de78155993b9-kube-api-access-4vshq" pod "2ecb79d2-6f5f-489f-b80d-de78155993b9" (UID: "2ecb79d2-6f5f-489f-b80d-de78155993b9"). InnerVolumeSpecName "kube-api-access-4vshq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:30.327951 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.327916 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/290fa234-8b22-4932-a683-a782790c388d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290fa234-8b22-4932-a683-a782790c388d-plugins-config-volume\") pod \"290fa234-8b22-4932-a683-a782790c388d\" (UID: \"290fa234-8b22-4932-a683-a782790c388d\") " Nov 04 01:36:30.328122 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.327995 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290fa234-8b22-4932-a683-a782790c388d-kube-api-access-xxqfx\" (UniqueName: \"kubernetes.io/projected/290fa234-8b22-4932-a683-a782790c388d-kube-api-access-xxqfx\") pod \"290fa234-8b22-4932-a683-a782790c388d\" (UID: \"290fa234-8b22-4932-a683-a782790c388d\") " Nov 04 01:36:30.328325 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.328299 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290fa234-8b22-4932-a683-a782790c388d-plugins-config-volume" pod "290fa234-8b22-4932-a683-a782790c388d" (UID: "290fa234-8b22-4932-a683-a782790c388d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:30.328498 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.328482 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vshq\" (UniqueName: \"kubernetes.io/projected/2ecb79d2-6f5f-489f-b80d-de78155993b9-kube-api-access-4vshq\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:30.328585 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.328499 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/290fa234-8b22-4932-a683-a782790c388d-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:30.330084 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.330059 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290fa234-8b22-4932-a683-a782790c388d-kube-api-access-xxqfx" pod "290fa234-8b22-4932-a683-a782790c388d" (UID: "290fa234-8b22-4932-a683-a782790c388d"). InnerVolumeSpecName "kube-api-access-xxqfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:30.380087 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.380052 5995 generic.go:356] "Generic (PLEG): container finished" podID="290fa234-8b22-4932-a683-a782790c388d" containerID="8f118cf833e12f434de0d4e1314c1e07a803dd3c6e6dce59eeafed00ce9581fd" exitCode=0 Nov 04 01:36:30.380267 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.380100 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" Nov 04 01:36:30.380267 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.380147 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" event={"ID":"290fa234-8b22-4932-a683-a782790c388d","Type":"ContainerDied","Data":"8f118cf833e12f434de0d4e1314c1e07a803dd3c6e6dce59eeafed00ce9581fd"} Nov 04 01:36:30.380267 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.380197 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx" event={"ID":"290fa234-8b22-4932-a683-a782790c388d","Type":"ContainerDied","Data":"c010e5df4502e621b9c48fcf3ce08330ecda054f600ec3625b77e131e872501b"} Nov 04 01:36:30.380267 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.380224 5995 scope.go:122] "RemoveContainer" containerID="8f118cf833e12f434de0d4e1314c1e07a803dd3c6e6dce59eeafed00ce9581fd" Nov 04 01:36:30.381947 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.381924 5995 generic.go:356] "Generic (PLEG): container finished" podID="2ecb79d2-6f5f-489f-b80d-de78155993b9" containerID="505324c77f260825daf96fe9f803b0c07a690177d6de53e49e2edd31c737280e" exitCode=0 Nov 04 01:36:30.382051 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.382031 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" event={"ID":"2ecb79d2-6f5f-489f-b80d-de78155993b9","Type":"ContainerDied","Data":"505324c77f260825daf96fe9f803b0c07a690177d6de53e49e2edd31c737280e"} Nov 04 01:36:30.382117 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.382092 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" event={"ID":"2ecb79d2-6f5f-489f-b80d-de78155993b9","Type":"ContainerDied","Data":"ebbde33b24f9aee1cd3c0e9b8a4692cdd60fc064b8395e7c9a14539564066b38"} Nov 04 01:36:30.382189 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.382171 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq" Nov 04 01:36:30.392198 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.392098 5995 scope.go:122] "RemoveContainer" containerID="8f118cf833e12f434de0d4e1314c1e07a803dd3c6e6dce59eeafed00ce9581fd" Nov 04 01:36:30.392502 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:30.392483 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f118cf833e12f434de0d4e1314c1e07a803dd3c6e6dce59eeafed00ce9581fd\": container with ID starting with 8f118cf833e12f434de0d4e1314c1e07a803dd3c6e6dce59eeafed00ce9581fd not found: ID does not exist" containerID="8f118cf833e12f434de0d4e1314c1e07a803dd3c6e6dce59eeafed00ce9581fd" Nov 04 01:36:30.392567 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.392512 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f118cf833e12f434de0d4e1314c1e07a803dd3c6e6dce59eeafed00ce9581fd"} err="failed to get container status \"8f118cf833e12f434de0d4e1314c1e07a803dd3c6e6dce59eeafed00ce9581fd\": rpc error: code = NotFound desc = could not find container \"8f118cf833e12f434de0d4e1314c1e07a803dd3c6e6dce59eeafed00ce9581fd\": container with ID starting with 8f118cf833e12f434de0d4e1314c1e07a803dd3c6e6dce59eeafed00ce9581fd not found: ID does not exist" Nov 04 01:36:30.392567 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.392531 5995 scope.go:122] "RemoveContainer" containerID="505324c77f260825daf96fe9f803b0c07a690177d6de53e49e2edd31c737280e" Nov 04 01:36:30.402545 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.402522 5995 scope.go:122] "RemoveContainer" containerID="505324c77f260825daf96fe9f803b0c07a690177d6de53e49e2edd31c737280e" Nov 04 01:36:30.402813 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:30.402794 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505324c77f260825daf96fe9f803b0c07a690177d6de53e49e2edd31c737280e\": container with ID starting with 505324c77f260825daf96fe9f803b0c07a690177d6de53e49e2edd31c737280e not found: ID does not exist" containerID="505324c77f260825daf96fe9f803b0c07a690177d6de53e49e2edd31c737280e" Nov 04 01:36:30.402878 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.402826 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505324c77f260825daf96fe9f803b0c07a690177d6de53e49e2edd31c737280e"} err="failed to get container status \"505324c77f260825daf96fe9f803b0c07a690177d6de53e49e2edd31c737280e\": rpc error: code = NotFound desc = could not find container \"505324c77f260825daf96fe9f803b0c07a690177d6de53e49e2edd31c737280e\": container with ID starting with 505324c77f260825daf96fe9f803b0c07a690177d6de53e49e2edd31c737280e not found: ID does not exist" Nov 04 01:36:30.430150 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.430114 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxqfx\" (UniqueName: \"kubernetes.io/projected/290fa234-8b22-4932-a683-a782790c388d-kube-api-access-xxqfx\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:30.536408 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.536291 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx"] Nov 04 01:36:30.542921 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.542855 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1baf54-5dd6c8447f-k8fgx"] Nov 04 01:36:30.645920 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.645882 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq"] Nov 04 01:36:30.649690 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:30.649665 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1baf54-666476dc6b-gj2pq"] Nov 04 01:36:31.674871 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:31.674826 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn" podUID="0957c760-9805-417a-ab0f-1289fb7a70ad" containerName="payload-processing" containerID="cri-o://9a1d67788156e949b6cc16bd2d6e7e4db0f7ac864313ed233cf0541bbde73562" gracePeriod=25 Nov 04 01:36:31.819812 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:31.819771 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt" podUID="48618274-e82a-4031-aff6-f2ff37064488" containerName="payload-pre-processing" containerID="cri-o://8b53f40eb5eddc2099492dd7096e103aad91c8e1038e7a01d614451b4dfe034b" gracePeriod=25 Nov 04 01:36:31.973396 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:31.973363 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn" Nov 04 01:36:32.077154 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.077115 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0957c760-9805-417a-ab0f-1289fb7a70ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0957c760-9805-417a-ab0f-1289fb7a70ad-plugins-config-volume\") pod \"0957c760-9805-417a-ab0f-1289fb7a70ad\" (UID: \"0957c760-9805-417a-ab0f-1289fb7a70ad\") " Nov 04 01:36:32.077396 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.077210 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0957c760-9805-417a-ab0f-1289fb7a70ad-kube-api-access-d8gfp\" (UniqueName: \"kubernetes.io/projected/0957c760-9805-417a-ab0f-1289fb7a70ad-kube-api-access-d8gfp\") pod \"0957c760-9805-417a-ab0f-1289fb7a70ad\" (UID: \"0957c760-9805-417a-ab0f-1289fb7a70ad\") " Nov 04 01:36:32.078290 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.078237 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0957c760-9805-417a-ab0f-1289fb7a70ad-plugins-config-volume" pod "0957c760-9805-417a-ab0f-1289fb7a70ad" (UID: "0957c760-9805-417a-ab0f-1289fb7a70ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:32.088602 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.088567 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0957c760-9805-417a-ab0f-1289fb7a70ad-kube-api-access-d8gfp" pod "0957c760-9805-417a-ab0f-1289fb7a70ad" (UID: "0957c760-9805-417a-ab0f-1289fb7a70ad"). InnerVolumeSpecName "kube-api-access-d8gfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:32.114898 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.114869 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt" Nov 04 01:36:32.134515 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.134479 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="290fa234-8b22-4932-a683-a782790c388d" path="/var/lib/kubelet/pods/290fa234-8b22-4932-a683-a782790c388d/volumes" Nov 04 01:36:32.135108 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.135088 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ecb79d2-6f5f-489f-b80d-de78155993b9" path="/var/lib/kubelet/pods/2ecb79d2-6f5f-489f-b80d-de78155993b9/volumes" Nov 04 01:36:32.178516 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.178486 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0957c760-9805-417a-ab0f-1289fb7a70ad-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:32.178516 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.178511 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8gfp\" (UniqueName: \"kubernetes.io/projected/0957c760-9805-417a-ab0f-1289fb7a70ad-kube-api-access-d8gfp\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:32.289492 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.281300 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48618274-e82a-4031-aff6-f2ff37064488-kube-api-access-mhq2l\" (UniqueName: \"kubernetes.io/projected/48618274-e82a-4031-aff6-f2ff37064488-kube-api-access-mhq2l\") pod \"48618274-e82a-4031-aff6-f2ff37064488\" (UID: \"48618274-e82a-4031-aff6-f2ff37064488\") " Nov 04 01:36:32.289492 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.281712 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48618274-e82a-4031-aff6-f2ff37064488-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48618274-e82a-4031-aff6-f2ff37064488-plugins-config-volume\") pod \"48618274-e82a-4031-aff6-f2ff37064488\" (UID: \"48618274-e82a-4031-aff6-f2ff37064488\") " Nov 04 01:36:32.341710 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.341664 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48618274-e82a-4031-aff6-f2ff37064488-plugins-config-volume" pod "48618274-e82a-4031-aff6-f2ff37064488" (UID: "48618274-e82a-4031-aff6-f2ff37064488"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:32.342263 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.342222 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48618274-e82a-4031-aff6-f2ff37064488-kube-api-access-mhq2l" pod "48618274-e82a-4031-aff6-f2ff37064488" (UID: "48618274-e82a-4031-aff6-f2ff37064488"). InnerVolumeSpecName "kube-api-access-mhq2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:32.399460 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.387138 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhq2l\" (UniqueName: \"kubernetes.io/projected/48618274-e82a-4031-aff6-f2ff37064488-kube-api-access-mhq2l\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:32.399460 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.387173 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48618274-e82a-4031-aff6-f2ff37064488-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:32.455184 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.455145 5995 generic.go:356] "Generic (PLEG): container finished" podID="48618274-e82a-4031-aff6-f2ff37064488" containerID="8b53f40eb5eddc2099492dd7096e103aad91c8e1038e7a01d614451b4dfe034b" exitCode=0 Nov 04 01:36:32.455532 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.455197 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt" Nov 04 01:36:32.456057 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.455216 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt" event={"ID":"48618274-e82a-4031-aff6-f2ff37064488","Type":"ContainerDied","Data":"8b53f40eb5eddc2099492dd7096e103aad91c8e1038e7a01d614451b4dfe034b"} Nov 04 01:36:32.456057 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.455820 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt" event={"ID":"48618274-e82a-4031-aff6-f2ff37064488","Type":"ContainerDied","Data":"578f7896fd2a6ed7dfc54f00256b8d4907b9b714ba21fe5412fec71a07b29dc4"} Nov 04 01:36:32.456057 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.455850 5995 scope.go:122] "RemoveContainer" containerID="8b53f40eb5eddc2099492dd7096e103aad91c8e1038e7a01d614451b4dfe034b" Nov 04 01:36:32.457662 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.457640 5995 generic.go:356] "Generic (PLEG): container finished" podID="0957c760-9805-417a-ab0f-1289fb7a70ad" containerID="9a1d67788156e949b6cc16bd2d6e7e4db0f7ac864313ed233cf0541bbde73562" exitCode=0 Nov 04 01:36:32.457838 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.457753 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn" event={"ID":"0957c760-9805-417a-ab0f-1289fb7a70ad","Type":"ContainerDied","Data":"9a1d67788156e949b6cc16bd2d6e7e4db0f7ac864313ed233cf0541bbde73562"} Nov 04 01:36:32.457838 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.457830 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn" event={"ID":"0957c760-9805-417a-ab0f-1289fb7a70ad","Type":"ContainerDied","Data":"2d782afaf70515011de59ddb1c6581d56c101113f7ef1922cd940180b00cb6f5"} Nov 04 01:36:32.458000 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.457979 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn" Nov 04 01:36:32.467865 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.467843 5995 scope.go:122] "RemoveContainer" containerID="8b53f40eb5eddc2099492dd7096e103aad91c8e1038e7a01d614451b4dfe034b" Nov 04 01:36:32.468186 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:32.468164 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b53f40eb5eddc2099492dd7096e103aad91c8e1038e7a01d614451b4dfe034b\": container with ID starting with 8b53f40eb5eddc2099492dd7096e103aad91c8e1038e7a01d614451b4dfe034b not found: ID does not exist" containerID="8b53f40eb5eddc2099492dd7096e103aad91c8e1038e7a01d614451b4dfe034b" Nov 04 01:36:32.468279 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.468198 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b53f40eb5eddc2099492dd7096e103aad91c8e1038e7a01d614451b4dfe034b"} err="failed to get container status \"8b53f40eb5eddc2099492dd7096e103aad91c8e1038e7a01d614451b4dfe034b\": rpc error: code = NotFound desc = could not find container \"8b53f40eb5eddc2099492dd7096e103aad91c8e1038e7a01d614451b4dfe034b\": container with ID starting with 8b53f40eb5eddc2099492dd7096e103aad91c8e1038e7a01d614451b4dfe034b not found: ID does not exist" Nov 04 01:36:32.468279 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.468222 5995 scope.go:122] "RemoveContainer" containerID="9a1d67788156e949b6cc16bd2d6e7e4db0f7ac864313ed233cf0541bbde73562" Nov 04 01:36:32.478008 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.477965 5995 scope.go:122] "RemoveContainer" containerID="9a1d67788156e949b6cc16bd2d6e7e4db0f7ac864313ed233cf0541bbde73562" Nov 04 01:36:32.478465 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:32.478399 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1d67788156e949b6cc16bd2d6e7e4db0f7ac864313ed233cf0541bbde73562\": container with ID starting with 9a1d67788156e949b6cc16bd2d6e7e4db0f7ac864313ed233cf0541bbde73562 not found: ID does not exist" containerID="9a1d67788156e949b6cc16bd2d6e7e4db0f7ac864313ed233cf0541bbde73562" Nov 04 01:36:32.478564 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.478453 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1d67788156e949b6cc16bd2d6e7e4db0f7ac864313ed233cf0541bbde73562"} err="failed to get container status \"9a1d67788156e949b6cc16bd2d6e7e4db0f7ac864313ed233cf0541bbde73562\": rpc error: code = NotFound desc = could not find container \"9a1d67788156e949b6cc16bd2d6e7e4db0f7ac864313ed233cf0541bbde73562\": container with ID starting with 9a1d67788156e949b6cc16bd2d6e7e4db0f7ac864313ed233cf0541bbde73562 not found: ID does not exist" Nov 04 01:36:32.610446 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.610343 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn"] Nov 04 01:36:32.615727 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.615674 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5520b3-78b5854954-4pltn"] Nov 04 01:36:32.717182 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.717148 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt"] Nov 04 01:36:32.724062 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:32.724036 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5520b3-599b6d8447-z2cnt"] Nov 04 01:36:34.129133 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:34.129095 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0957c760-9805-417a-ab0f-1289fb7a70ad" path="/var/lib/kubelet/pods/0957c760-9805-417a-ab0f-1289fb7a70ad/volumes" Nov 04 01:36:34.129982 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:34.129507 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48618274-e82a-4031-aff6-f2ff37064488" path="/var/lib/kubelet/pods/48618274-e82a-4031-aff6-f2ff37064488/volumes" Nov 04 01:36:37.719555 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:37.719517 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:36:37.759903 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:37.759869 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bec17d7cdfd4: secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:37.759903 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:37.759900 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz: secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:37.760101 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:37.759961 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials podName:7650b8ae-725e-470e-8960-5053aa977ea0 nodeName:}" failed. No retries permitted until 2026-11-04 01:36:53.759946884 +0000 UTC m=+1563.791680256 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials") pod "payload-processing-external-model-6fddb87f65-hxqhz" (UID: "7650b8ae-725e-470e-8960-5053aa977ea0") : secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:47.143199 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:47.143159 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw"] Nov 04 01:36:47.143678 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:47.143438 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" podUID="8fa987f7-090b-477e-9779-008bcc81642c" containerName="istio-proxy" containerID="cri-o://5abe425217937dd6d60a7af1fa9fc427474ddfa5ae9df3865bff71b465337f63" gracePeriod=30 Nov 04 01:36:47.720185 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:47.720141 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:36:49.643713 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:49.643679 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:36:52.153326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.153247 5995 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal" podUID="5be03f35a536944b4cf2058a360cdcb6" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:36:52.298270 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.298230 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n"] Nov 04 01:36:52.298892 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.298872 5995 cpu_manager.go:414] "containerMap: removing container" podUID="2ecb79d2-6f5f-489f-b80d-de78155993b9" containerName="payload-processing" Nov 04 01:36:52.299009 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.298894 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ecb79d2-6f5f-489f-b80d-de78155993b9" containerName="payload-processing" Nov 04 01:36:52.299009 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.298936 5995 cpu_manager.go:414] "containerMap: removing container" podUID="0957c760-9805-417a-ab0f-1289fb7a70ad" containerName="payload-processing" Nov 04 01:36:52.299009 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.298945 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0957c760-9805-417a-ab0f-1289fb7a70ad" containerName="payload-processing" Nov 04 01:36:52.299009 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.298962 5995 cpu_manager.go:414] "containerMap: removing container" podUID="4bbe9a41-a5fc-4ee3-98e2-18be1232cffb" containerName="main" Nov 04 01:36:52.299009 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.298969 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bbe9a41-a5fc-4ee3-98e2-18be1232cffb" containerName="main" Nov 04 01:36:52.299009 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.298988 5995 cpu_manager.go:414] "containerMap: removing container" podUID="3443e17f-4370-4c7e-8d10-8a79d491dca4" containerName="payload-processing" Nov 04 01:36:52.299009 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.298995 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3443e17f-4370-4c7e-8d10-8a79d491dca4" containerName="payload-processing" Nov 04 01:36:52.299343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.299012 5995 cpu_manager.go:414] "containerMap: removing container" podUID="bdd33b9f-6727-4178-b340-6845fe234a1d" containerName="revoke-keys" Nov 04 01:36:52.299343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.299020 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd33b9f-6727-4178-b340-6845fe234a1d" containerName="revoke-keys" Nov 04 01:36:52.299343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.299048 5995 cpu_manager.go:414] "containerMap: removing container" podUID="48618274-e82a-4031-aff6-f2ff37064488" containerName="payload-pre-processing" Nov 04 01:36:52.299343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.299055 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48618274-e82a-4031-aff6-f2ff37064488" containerName="payload-pre-processing" Nov 04 01:36:52.299343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.299072 5995 cpu_manager.go:414] "containerMap: removing container" podUID="290fa234-8b22-4932-a683-a782790c388d" containerName="payload-pre-processing" Nov 04 01:36:52.299343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.299079 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290fa234-8b22-4932-a683-a782790c388d" containerName="payload-pre-processing" Nov 04 01:36:52.299343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.299219 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="48618274-e82a-4031-aff6-f2ff37064488" containerName="payload-pre-processing" Nov 04 01:36:52.299343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.299235 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="290fa234-8b22-4932-a683-a782790c388d" containerName="payload-pre-processing" Nov 04 01:36:52.299343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.299248 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ecb79d2-6f5f-489f-b80d-de78155993b9" containerName="payload-processing" Nov 04 01:36:52.299343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.299262 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="0957c760-9805-417a-ab0f-1289fb7a70ad" containerName="payload-processing" Nov 04 01:36:52.299343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.299271 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bbe9a41-a5fc-4ee3-98e2-18be1232cffb" containerName="main" Nov 04 01:36:52.299343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.299282 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="3443e17f-4370-4c7e-8d10-8a79d491dca4" containerName="payload-processing" Nov 04 01:36:52.299343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.299293 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd33b9f-6727-4178-b340-6845fe234a1d" containerName="revoke-keys" Nov 04 01:36:52.303440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.303156 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.352703 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.352670 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-1baf54-openshift-default-dockercfg-hnfrz\"" Nov 04 01:36:52.404307 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.404230 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n"] Nov 04 01:36:52.406294 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.406260 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istiod-ca-cert\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.406486 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.406359 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-token\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.406486 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.406404 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9mzd\" (UniqueName: \"kubernetes.io/projected/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-kube-api-access-n9mzd\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.406486 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.406443 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-data\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.406486 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.406487 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-podinfo\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.406736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.406523 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-workload-socket\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.406736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.406540 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-envoy\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.406736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.406567 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-workload-certs\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.406736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.406605 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-credential-socket\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.432973 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.432941 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:36:52.507676 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.507639 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-workload-certs\") pod \"8fa987f7-090b-477e-9779-008bcc81642c\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " Nov 04 01:36:52.507872 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.507686 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa987f7-090b-477e-9779-008bcc81642c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa987f7-090b-477e-9779-008bcc81642c-istiod-ca-cert\") pod \"8fa987f7-090b-477e-9779-008bcc81642c\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " Nov 04 01:36:52.507872 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.507720 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa987f7-090b-477e-9779-008bcc81642c-istio-token\" (UniqueName: \"kubernetes.io/projected/8fa987f7-090b-477e-9779-008bcc81642c-istio-token\") pod \"8fa987f7-090b-477e-9779-008bcc81642c\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " Nov 04 01:36:52.507872 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.507771 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-credential-socket\") pod \"8fa987f7-090b-477e-9779-008bcc81642c\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " Nov 04 01:36:52.508052 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.507967 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-workload-socket\") pod \"8fa987f7-090b-477e-9779-008bcc81642c\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " Nov 04 01:36:52.508052 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.508023 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fa987f7-090b-477e-9779-008bcc81642c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa987f7-090b-477e-9779-008bcc81642c-istio-podinfo\") pod \"8fa987f7-090b-477e-9779-008bcc81642c\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " Nov 04 01:36:52.508052 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.507965 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-workload-certs" pod "8fa987f7-090b-477e-9779-008bcc81642c" (UID: "8fa987f7-090b-477e-9779-008bcc81642c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:52.508206 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.508052 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-credential-socket" pod "8fa987f7-090b-477e-9779-008bcc81642c" (UID: "8fa987f7-090b-477e-9779-008bcc81642c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:52.508206 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.508109 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-istio-data\") pod \"8fa987f7-090b-477e-9779-008bcc81642c\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " Nov 04 01:36:52.508206 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.508170 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-workload-socket" pod "8fa987f7-090b-477e-9779-008bcc81642c" (UID: "8fa987f7-090b-477e-9779-008bcc81642c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:52.508206 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.508186 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-istio-envoy\") pod \"8fa987f7-090b-477e-9779-008bcc81642c\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " Nov 04 01:36:52.508403 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.508207 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa987f7-090b-477e-9779-008bcc81642c-istiod-ca-cert" pod "8fa987f7-090b-477e-9779-008bcc81642c" (UID: "8fa987f7-090b-477e-9779-008bcc81642c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:52.508403 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.508236 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa987f7-090b-477e-9779-008bcc81642c-kube-api-access-g2jpq\" (UniqueName: \"kubernetes.io/projected/8fa987f7-090b-477e-9779-008bcc81642c-kube-api-access-g2jpq\") pod \"8fa987f7-090b-477e-9779-008bcc81642c\" (UID: \"8fa987f7-090b-477e-9779-008bcc81642c\") " Nov 04 01:36:52.508545 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.508403 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-workload-socket\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.508545 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.508453 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-envoy\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.508545 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.508453 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-istio-data" pod "8fa987f7-090b-477e-9779-008bcc81642c" (UID: "8fa987f7-090b-477e-9779-008bcc81642c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:52.508545 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.508530 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-workload-certs\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.508736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.508600 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-credential-socket\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.508736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.508716 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istiod-ca-cert\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.508848 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.508818 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-token\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.508904 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.508860 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-workload-socket\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.508904 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.508884 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9mzd\" (UniqueName: \"kubernetes.io/projected/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-kube-api-access-n9mzd\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.509020 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.508925 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-data\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.509020 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.509012 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-podinfo\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.509145 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.509113 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-credential-socket\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.509211 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.509116 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:52.509211 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.509167 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:52.509211 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.509187 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:52.509211 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.509204 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa987f7-090b-477e-9779-008bcc81642c-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:52.509435 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.509219 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:52.510314 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.510263 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-data\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.510586 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.510555 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa987f7-090b-477e-9779-008bcc81642c-istio-token" pod "8fa987f7-090b-477e-9779-008bcc81642c" (UID: "8fa987f7-090b-477e-9779-008bcc81642c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:52.510698 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.510613 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-workload-certs\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.510698 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.510674 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istiod-ca-cert\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.511133 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.511097 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa987f7-090b-477e-9779-008bcc81642c-kube-api-access-g2jpq" pod "8fa987f7-090b-477e-9779-008bcc81642c" (UID: "8fa987f7-090b-477e-9779-008bcc81642c"). InnerVolumeSpecName "kube-api-access-g2jpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:52.511251 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.511172 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-envoy\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.511251 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.511214 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fa987f7-090b-477e-9779-008bcc81642c-istio-podinfo" pod "8fa987f7-090b-477e-9779-008bcc81642c" (UID: "8fa987f7-090b-477e-9779-008bcc81642c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:36:52.511379 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.511371 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-istio-envoy" pod "8fa987f7-090b-477e-9779-008bcc81642c" (UID: "8fa987f7-090b-477e-9779-008bcc81642c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:52.512066 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.512040 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-podinfo\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.567690 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.567662 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-token\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.567847 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.567780 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9mzd\" (UniqueName: \"kubernetes.io/projected/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-kube-api-access-n9mzd\") pod \"e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.610110 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.610077 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa987f7-090b-477e-9779-008bcc81642c-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:52.610110 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.610106 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2jpq\" (UniqueName: \"kubernetes.io/projected/8fa987f7-090b-477e-9779-008bcc81642c-kube-api-access-g2jpq\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:52.610110 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.610115 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa987f7-090b-477e-9779-008bcc81642c-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:52.610338 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.610123 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa987f7-090b-477e-9779-008bcc81642c-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:52.618772 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.618744 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:52.671489 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.666342 5995 generic.go:356] "Generic (PLEG): container finished" podID="8fa987f7-090b-477e-9779-008bcc81642c" containerID="5abe425217937dd6d60a7af1fa9fc427474ddfa5ae9df3865bff71b465337f63" exitCode=0 Nov 04 01:36:52.671489 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.666519 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" event={"ID":"8fa987f7-090b-477e-9779-008bcc81642c","Type":"ContainerDied","Data":"5abe425217937dd6d60a7af1fa9fc427474ddfa5ae9df3865bff71b465337f63"} Nov 04 01:36:52.671489 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.666574 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" event={"ID":"8fa987f7-090b-477e-9779-008bcc81642c","Type":"ContainerDied","Data":"7454314aa9247fabd1e19599e94f707a4479e0239a25a804b103a463bcdbc646"} Nov 04 01:36:52.671489 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.666604 5995 scope.go:122] "RemoveContainer" containerID="5abe425217937dd6d60a7af1fa9fc427474ddfa5ae9df3865bff71b465337f63" Nov 04 01:36:52.671489 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.666766 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw" Nov 04 01:36:52.696134 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.694626 5995 scope.go:122] "RemoveContainer" containerID="71c6227ee4977db1e94a7f1c18b2b2abf859a564cfc7dbe35ef71e8152562f55" Nov 04 01:36:52.729232 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.727685 5995 scope.go:122] "RemoveContainer" containerID="5abe425217937dd6d60a7af1fa9fc427474ddfa5ae9df3865bff71b465337f63" Nov 04 01:36:52.729681 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:52.729472 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5abe425217937dd6d60a7af1fa9fc427474ddfa5ae9df3865bff71b465337f63\": container with ID starting with 5abe425217937dd6d60a7af1fa9fc427474ddfa5ae9df3865bff71b465337f63 not found: ID does not exist" containerID="5abe425217937dd6d60a7af1fa9fc427474ddfa5ae9df3865bff71b465337f63" Nov 04 01:36:52.729681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.729504 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5abe425217937dd6d60a7af1fa9fc427474ddfa5ae9df3865bff71b465337f63"} err="failed to get container status \"5abe425217937dd6d60a7af1fa9fc427474ddfa5ae9df3865bff71b465337f63\": rpc error: code = NotFound desc = could not find container \"5abe425217937dd6d60a7af1fa9fc427474ddfa5ae9df3865bff71b465337f63\": container with ID starting with 5abe425217937dd6d60a7af1fa9fc427474ddfa5ae9df3865bff71b465337f63 not found: ID does not exist" Nov 04 01:36:52.729681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.729522 5995 scope.go:122] "RemoveContainer" containerID="71c6227ee4977db1e94a7f1c18b2b2abf859a564cfc7dbe35ef71e8152562f55" Nov 04 01:36:52.730003 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:52.729927 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c6227ee4977db1e94a7f1c18b2b2abf859a564cfc7dbe35ef71e8152562f55\": container with ID starting with 71c6227ee4977db1e94a7f1c18b2b2abf859a564cfc7dbe35ef71e8152562f55 not found: ID does not exist" containerID="71c6227ee4977db1e94a7f1c18b2b2abf859a564cfc7dbe35ef71e8152562f55" Nov 04 01:36:52.730003 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.729952 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c6227ee4977db1e94a7f1c18b2b2abf859a564cfc7dbe35ef71e8152562f55"} err="failed to get container status \"71c6227ee4977db1e94a7f1c18b2b2abf859a564cfc7dbe35ef71e8152562f55\": rpc error: code = NotFound desc = could not find container \"71c6227ee4977db1e94a7f1c18b2b2abf859a564cfc7dbe35ef71e8152562f55\": container with ID starting with 71c6227ee4977db1e94a7f1c18b2b2abf859a564cfc7dbe35ef71e8152562f55 not found: ID does not exist" Nov 04 01:36:52.837879 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.837819 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw"] Nov 04 01:36:52.841702 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:52.841678 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5520b3-openshift-default-6848bdfc-l5vpw"] Nov 04 01:36:53.092257 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:53.092203 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n"] Nov 04 01:36:53.096782 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:36:53.096748 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c37edb3_f54c_4a56_8ce1_08109fe90b2f.slice/crio-1a3ef7489d9aa7ccf64ee125727643993d2f74947eceb3fb0bce7df5935c7494 WatchSource:0}: Error finding container 1a3ef7489d9aa7ccf64ee125727643993d2f74947eceb3fb0bce7df5935c7494: Status 404 returned error can't find the container with id 1a3ef7489d9aa7ccf64ee125727643993d2f74947eceb3fb0bce7df5935c7494 Nov 04 01:36:53.100608 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:53.100564 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:36:53.100723 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:53.100657 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:36:53.100723 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:53.100698 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:36:53.698657 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:53.698611 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" event={"ID":"6c37edb3-f54c-4a56-8ce1-08109fe90b2f","Type":"ContainerStarted","Data":"e69a5125b1264cc31127ddbaff1b505959addf042984fa746df7ace1b2cbef66"} Nov 04 01:36:53.699149 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:53.698679 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" event={"ID":"6c37edb3-f54c-4a56-8ce1-08109fe90b2f","Type":"ContainerStarted","Data":"1a3ef7489d9aa7ccf64ee125727643993d2f74947eceb3fb0bce7df5935c7494"} Nov 04 01:36:53.814168 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:53.814107 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" podStartSLOduration=1.8140936920000001 podStartE2EDuration="1.814093692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:36:53.812111044 +0000 UTC m=+1563.843844445" watchObservedRunningTime="2026-11-04 01:36:53.814093692 +0000 UTC m=+1563.845827082" Nov 04 01:36:53.825279 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:53.825248 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bec17d7cdfd4: secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:53.825279 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:53.825277 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz: secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:53.825496 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:36:53.825332 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials podName:7650b8ae-725e-470e-8960-5053aa977ea0 nodeName:}" failed. No retries permitted until 2026-11-04 01:37:25.825317045 +0000 UTC m=+1595.857050413 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials") pod "payload-processing-external-model-6fddb87f65-hxqhz" (UID: "7650b8ae-725e-470e-8960-5053aa977ea0") : secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:36:54.131276 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:54.131193 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa987f7-090b-477e-9779-008bcc81642c" path="/var/lib/kubelet/pods/8fa987f7-090b-477e-9779-008bcc81642c/volumes" Nov 04 01:36:54.629441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:54.626497 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:36:54.629441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:54.627846 5995 patch_prober.go:29] interesting pod/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:36:54.629441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:54.627903 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" podUID="6c37edb3-f54c-4a56-8ce1-08109fe90b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:36:55.205601 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:55.204710 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq"] Nov 04 01:36:55.619453 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:55.619336 5995 patch_prober.go:29] interesting pod/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:36:55.638352 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:55.637678 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" podUID="6c37edb3-f54c-4a56-8ce1-08109fe90b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:36:56.622028 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:56.621985 5995 patch_prober.go:29] interesting pod/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:36:56.622675 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:56.622640 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" podUID="6c37edb3-f54c-4a56-8ce1-08109fe90b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:36:57.619176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:57.619138 5995 patch_prober.go:29] interesting pod/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:36:57.619380 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:57.619190 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" podUID="6c37edb3-f54c-4a56-8ce1-08109fe90b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:36:57.719336 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:57.719292 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:36:58.619457 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:58.619395 5995 patch_prober.go:29] interesting pod/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:36:58.619641 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:58.619498 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" podUID="6c37edb3-f54c-4a56-8ce1-08109fe90b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:36:59.619094 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:59.619054 5995 patch_prober.go:29] interesting pod/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:36:59.619590 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:36:59.619109 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" podUID="6c37edb3-f54c-4a56-8ce1-08109fe90b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:37:00.205631 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:00.205581 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq" podUID="9fce5c04-9c5c-42f5-bfda-e94dbae0b266" containerName="payload-pre-processing" containerID="cri-o://5ac3ab0c9707ad0c67bd0cd3a01eb55793d7d64b6f05d254e8141f53f8f3aad3" gracePeriod=25 Nov 04 01:37:00.674874 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:00.674841 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq" Nov 04 01:37:00.887489 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:00.867483 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fce5c04-9c5c-42f5-bfda-e94dbae0b266-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fce5c04-9c5c-42f5-bfda-e94dbae0b266-plugins-config-volume\") pod \"9fce5c04-9c5c-42f5-bfda-e94dbae0b266\" (UID: \"9fce5c04-9c5c-42f5-bfda-e94dbae0b266\") " Nov 04 01:37:00.887489 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:00.867567 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fce5c04-9c5c-42f5-bfda-e94dbae0b266-kube-api-access-bp8cp\" (UniqueName: \"kubernetes.io/projected/9fce5c04-9c5c-42f5-bfda-e94dbae0b266-kube-api-access-bp8cp\") pod \"9fce5c04-9c5c-42f5-bfda-e94dbae0b266\" (UID: \"9fce5c04-9c5c-42f5-bfda-e94dbae0b266\") " Nov 04 01:37:00.887489 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:00.881623 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fce5c04-9c5c-42f5-bfda-e94dbae0b266-plugins-config-volume" pod "9fce5c04-9c5c-42f5-bfda-e94dbae0b266" (UID: "9fce5c04-9c5c-42f5-bfda-e94dbae0b266"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:37:00.917480 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:00.917406 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fce5c04-9c5c-42f5-bfda-e94dbae0b266-kube-api-access-bp8cp" pod "9fce5c04-9c5c-42f5-bfda-e94dbae0b266" (UID: "9fce5c04-9c5c-42f5-bfda-e94dbae0b266"). InnerVolumeSpecName "kube-api-access-bp8cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:37:00.935621 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:00.935572 5995 generic.go:356] "Generic (PLEG): container finished" podID="9fce5c04-9c5c-42f5-bfda-e94dbae0b266" containerID="5ac3ab0c9707ad0c67bd0cd3a01eb55793d7d64b6f05d254e8141f53f8f3aad3" exitCode=0 Nov 04 01:37:00.935791 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:00.935690 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq" Nov 04 01:37:00.935791 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:00.935716 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq" event={"ID":"9fce5c04-9c5c-42f5-bfda-e94dbae0b266","Type":"ContainerDied","Data":"5ac3ab0c9707ad0c67bd0cd3a01eb55793d7d64b6f05d254e8141f53f8f3aad3"} Nov 04 01:37:00.935791 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:00.935774 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq" event={"ID":"9fce5c04-9c5c-42f5-bfda-e94dbae0b266","Type":"ContainerDied","Data":"e99904f1bd1a9cd7ad368c1fee3271c91e1319e699f3740a8ba4e3442325e4dd"} Nov 04 01:37:00.935954 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:00.935799 5995 scope.go:122] "RemoveContainer" containerID="5ac3ab0c9707ad0c67bd0cd3a01eb55793d7d64b6f05d254e8141f53f8f3aad3" Nov 04 01:37:00.948893 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:00.948877 5995 scope.go:122] "RemoveContainer" containerID="5ac3ab0c9707ad0c67bd0cd3a01eb55793d7d64b6f05d254e8141f53f8f3aad3" Nov 04 01:37:00.949683 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:37:00.949499 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac3ab0c9707ad0c67bd0cd3a01eb55793d7d64b6f05d254e8141f53f8f3aad3\": container with ID starting with 5ac3ab0c9707ad0c67bd0cd3a01eb55793d7d64b6f05d254e8141f53f8f3aad3 not found: ID does not exist" containerID="5ac3ab0c9707ad0c67bd0cd3a01eb55793d7d64b6f05d254e8141f53f8f3aad3" Nov 04 01:37:00.949683 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:00.949537 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac3ab0c9707ad0c67bd0cd3a01eb55793d7d64b6f05d254e8141f53f8f3aad3"} err="failed to get container status \"5ac3ab0c9707ad0c67bd0cd3a01eb55793d7d64b6f05d254e8141f53f8f3aad3\": rpc error: code = NotFound desc = could not find container \"5ac3ab0c9707ad0c67bd0cd3a01eb55793d7d64b6f05d254e8141f53f8f3aad3\": container with ID starting with 5ac3ab0c9707ad0c67bd0cd3a01eb55793d7d64b6f05d254e8141f53f8f3aad3 not found: ID does not exist" Nov 04 01:37:00.969497 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:00.969464 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fce5c04-9c5c-42f5-bfda-e94dbae0b266-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:00.969497 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:00.969498 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp8cp\" (UniqueName: \"kubernetes.io/projected/9fce5c04-9c5c-42f5-bfda-e94dbae0b266-kube-api-access-bp8cp\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:00.990446 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:00.990398 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:37:00.990877 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:00.990854 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:37:00.991819 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:00.991801 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:37:01.006439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:01.000253 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:37:01.097430 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:01.097383 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq"] Nov 04 01:37:01.101530 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:01.101500 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cae0ab-7bc4798f9-7qbwq"] Nov 04 01:37:02.163438 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:02.153509 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fce5c04-9c5c-42f5-bfda-e94dbae0b266" path="/var/lib/kubelet/pods/9fce5c04-9c5c-42f5-bfda-e94dbae0b266/volumes" Nov 04 01:37:07.719745 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:07.719703 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:37:09.123260 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:09.123221 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:37:09.399453 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:09.399351 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n"] Nov 04 01:37:09.399973 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:09.399946 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" podUID="6c37edb3-f54c-4a56-8ce1-08109fe90b2f" containerName="istio-proxy" containerID="cri-o://e69a5125b1264cc31127ddbaff1b505959addf042984fa746df7ace1b2cbef66" gracePeriod=30 Nov 04 01:37:14.689688 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.689660 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:37:14.861317 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.861280 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-podinfo\") pod \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " Nov 04 01:37:14.861541 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.861511 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istiod-ca-cert\") pod \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " Nov 04 01:37:14.861623 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.861551 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-credential-socket\") pod \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " Nov 04 01:37:14.861674 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.861641 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-token\" (UniqueName: \"kubernetes.io/projected/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-token\") pod \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " Nov 04 01:37:14.861728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.861675 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-envoy\") pod \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " Nov 04 01:37:14.861728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.861714 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-workload-certs\") pod \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " Nov 04 01:37:14.861825 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.861747 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-data\") pod \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " Nov 04 01:37:14.861825 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.861811 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-kube-api-access-n9mzd\" (UniqueName: \"kubernetes.io/projected/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-kube-api-access-n9mzd\") pod \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " Nov 04 01:37:14.861938 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.861904 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-credential-socket" pod "6c37edb3-f54c-4a56-8ce1-08109fe90b2f" (UID: "6c37edb3-f54c-4a56-8ce1-08109fe90b2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:37:14.861992 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.861930 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istiod-ca-cert" pod "6c37edb3-f54c-4a56-8ce1-08109fe90b2f" (UID: "6c37edb3-f54c-4a56-8ce1-08109fe90b2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:37:14.861992 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.861957 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-workload-socket\") pod \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\" (UID: \"6c37edb3-f54c-4a56-8ce1-08109fe90b2f\") " Nov 04 01:37:14.862165 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.862126 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-data" pod "6c37edb3-f54c-4a56-8ce1-08109fe90b2f" (UID: "6c37edb3-f54c-4a56-8ce1-08109fe90b2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:37:14.862575 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.862519 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-workload-socket" pod "6c37edb3-f54c-4a56-8ce1-08109fe90b2f" (UID: "6c37edb3-f54c-4a56-8ce1-08109fe90b2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:37:14.862575 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.862531 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:14.862575 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.862552 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:14.862575 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.862568 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:14.863555 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.863517 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-workload-certs" pod "6c37edb3-f54c-4a56-8ce1-08109fe90b2f" (UID: "6c37edb3-f54c-4a56-8ce1-08109fe90b2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:37:14.864180 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.864157 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-podinfo" pod "6c37edb3-f54c-4a56-8ce1-08109fe90b2f" (UID: "6c37edb3-f54c-4a56-8ce1-08109fe90b2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:37:14.864553 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.864530 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-kube-api-access-n9mzd" pod "6c37edb3-f54c-4a56-8ce1-08109fe90b2f" (UID: "6c37edb3-f54c-4a56-8ce1-08109fe90b2f"). InnerVolumeSpecName "kube-api-access-n9mzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:37:14.865033 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.865007 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-token" pod "6c37edb3-f54c-4a56-8ce1-08109fe90b2f" (UID: "6c37edb3-f54c-4a56-8ce1-08109fe90b2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:37:14.865250 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.865226 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-envoy" pod "6c37edb3-f54c-4a56-8ce1-08109fe90b2f" (UID: "6c37edb3-f54c-4a56-8ce1-08109fe90b2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:37:14.963577 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.963539 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:14.963577 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.963567 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:14.963577 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.963576 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:14.963577 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.963584 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:14.963577 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.963593 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:14.963898 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:14.963603 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9mzd\" (UniqueName: \"kubernetes.io/projected/6c37edb3-f54c-4a56-8ce1-08109fe90b2f-kube-api-access-n9mzd\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:15.087857 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:15.087819 5995 generic.go:356] "Generic (PLEG): container finished" podID="6c37edb3-f54c-4a56-8ce1-08109fe90b2f" containerID="e69a5125b1264cc31127ddbaff1b505959addf042984fa746df7ace1b2cbef66" exitCode=0 Nov 04 01:37:15.088044 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:15.087867 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" Nov 04 01:37:15.088044 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:15.087874 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" event={"ID":"6c37edb3-f54c-4a56-8ce1-08109fe90b2f","Type":"ContainerDied","Data":"e69a5125b1264cc31127ddbaff1b505959addf042984fa746df7ace1b2cbef66"} Nov 04 01:37:15.088044 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:15.087927 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n" event={"ID":"6c37edb3-f54c-4a56-8ce1-08109fe90b2f","Type":"ContainerDied","Data":"1a3ef7489d9aa7ccf64ee125727643993d2f74947eceb3fb0bce7df5935c7494"} Nov 04 01:37:15.088044 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:15.087954 5995 scope.go:122] "RemoveContainer" containerID="e69a5125b1264cc31127ddbaff1b505959addf042984fa746df7ace1b2cbef66" Nov 04 01:37:15.101345 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:15.101322 5995 scope.go:122] "RemoveContainer" containerID="e69a5125b1264cc31127ddbaff1b505959addf042984fa746df7ace1b2cbef66" Nov 04 01:37:15.101611 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:37:15.101590 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69a5125b1264cc31127ddbaff1b505959addf042984fa746df7ace1b2cbef66\": container with ID starting with e69a5125b1264cc31127ddbaff1b505959addf042984fa746df7ace1b2cbef66 not found: ID does not exist" containerID="e69a5125b1264cc31127ddbaff1b505959addf042984fa746df7ace1b2cbef66" Nov 04 01:37:15.101696 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:15.101618 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69a5125b1264cc31127ddbaff1b505959addf042984fa746df7ace1b2cbef66"} err="failed to get container status \"e69a5125b1264cc31127ddbaff1b505959addf042984fa746df7ace1b2cbef66\": rpc error: code = NotFound desc = could not find container \"e69a5125b1264cc31127ddbaff1b505959addf042984fa746df7ace1b2cbef66\": container with ID starting with e69a5125b1264cc31127ddbaff1b505959addf042984fa746df7ace1b2cbef66 not found: ID does not exist" Nov 04 01:37:15.245205 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:15.245089 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n"] Nov 04 01:37:15.291656 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:15.291620 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1baf54-openshift-default-84dc6cddcc-2zg9n"] Nov 04 01:37:16.130525 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:16.130487 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c37edb3-f54c-4a56-8ce1-08109fe90b2f" path="/var/lib/kubelet/pods/6c37edb3-f54c-4a56-8ce1-08109fe90b2f/volumes" Nov 04 01:37:17.780914 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:17.780883 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6"] Nov 04 01:37:17.781343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:17.781088 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" podUID="cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" containerName="istio-proxy" containerID="cri-o://3727b1294557a5d235d3e7c5430e36331cb9acfc29fa0eaeeb13e39db390d5f3" gracePeriod=30 Nov 04 01:37:18.564801 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:18.564760 5995 patch_prober.go:29] interesting pod/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:37:18.564979 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:18.564823 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" podUID="cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:37:19.123079 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.123048 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:37:19.368369 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.368335 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt"] Nov 04 01:37:19.369030 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.369011 5995 cpu_manager.go:414] "containerMap: removing container" podUID="8fa987f7-090b-477e-9779-008bcc81642c" containerName="istio-proxy" Nov 04 01:37:19.369116 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.369033 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa987f7-090b-477e-9779-008bcc81642c" containerName="istio-proxy" Nov 04 01:37:19.369116 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.369054 5995 cpu_manager.go:414] "containerMap: removing container" podUID="8fa987f7-090b-477e-9779-008bcc81642c" containerName="istio-proxy" Nov 04 01:37:19.369116 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.369062 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa987f7-090b-477e-9779-008bcc81642c" containerName="istio-proxy" Nov 04 01:37:19.369116 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.369103 5995 cpu_manager.go:414] "containerMap: removing container" podUID="6c37edb3-f54c-4a56-8ce1-08109fe90b2f" containerName="istio-proxy" Nov 04 01:37:19.369116 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.369111 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c37edb3-f54c-4a56-8ce1-08109fe90b2f" containerName="istio-proxy" Nov 04 01:37:19.369295 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.369127 5995 cpu_manager.go:414] "containerMap: removing container" podUID="9fce5c04-9c5c-42f5-bfda-e94dbae0b266" containerName="payload-pre-processing" Nov 04 01:37:19.369295 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.369134 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fce5c04-9c5c-42f5-bfda-e94dbae0b266" containerName="payload-pre-processing" Nov 04 01:37:19.369295 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.369270 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa987f7-090b-477e-9779-008bcc81642c" containerName="istio-proxy" Nov 04 01:37:19.369295 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.369286 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa987f7-090b-477e-9779-008bcc81642c" containerName="istio-proxy" Nov 04 01:37:19.369409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.369300 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c37edb3-f54c-4a56-8ce1-08109fe90b2f" containerName="istio-proxy" Nov 04 01:37:19.369409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.369307 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fce5c04-9c5c-42f5-bfda-e94dbae0b266" containerName="payload-pre-processing" Nov 04 01:37:19.374549 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.374492 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt" Nov 04 01:37:19.424410 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.424371 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:37:19.460324 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.460284 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77044b62-3256-44be-b8d7-245046a12ff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ad94dfa-zw5kt\" (UID: \"77044b62-3256-44be-b8d7-245046a12ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt" Nov 04 01:37:19.460557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.460539 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77044b62-3256-44be-b8d7-245046a12ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ad94dfa-zw5kt\" (UID: \"77044b62-3256-44be-b8d7-245046a12ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt" Nov 04 01:37:19.460638 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.460576 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnc7w\" (UniqueName: \"kubernetes.io/projected/77044b62-3256-44be-b8d7-245046a12ff1-kube-api-access-gnc7w\") pod \"maas-api-revoke-sub-3ad94dfa-zw5kt\" (UID: \"77044b62-3256-44be-b8d7-245046a12ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt" Nov 04 01:37:19.471262 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.471231 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:37:19.471409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.471282 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:37:19.471409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.471287 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w799x\"" Nov 04 01:37:19.474514 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.474494 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt"] Nov 04 01:37:19.562162 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.562125 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77044b62-3256-44be-b8d7-245046a12ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ad94dfa-zw5kt\" (UID: \"77044b62-3256-44be-b8d7-245046a12ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt" Nov 04 01:37:19.563197 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.563163 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnc7w\" (UniqueName: \"kubernetes.io/projected/77044b62-3256-44be-b8d7-245046a12ff1-kube-api-access-gnc7w\") pod \"maas-api-revoke-sub-3ad94dfa-zw5kt\" (UID: \"77044b62-3256-44be-b8d7-245046a12ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt" Nov 04 01:37:19.563338 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.563095 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77044b62-3256-44be-b8d7-245046a12ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ad94dfa-zw5kt\" (UID: \"77044b62-3256-44be-b8d7-245046a12ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt" Nov 04 01:37:19.563338 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.563260 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77044b62-3256-44be-b8d7-245046a12ff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ad94dfa-zw5kt\" (UID: \"77044b62-3256-44be-b8d7-245046a12ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt" Nov 04 01:37:19.566366 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.566341 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77044b62-3256-44be-b8d7-245046a12ff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ad94dfa-zw5kt\" (UID: \"77044b62-3256-44be-b8d7-245046a12ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt" Nov 04 01:37:19.620988 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.620960 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnc7w\" (UniqueName: \"kubernetes.io/projected/77044b62-3256-44be-b8d7-245046a12ff1-kube-api-access-gnc7w\") pod \"maas-api-revoke-sub-3ad94dfa-zw5kt\" (UID: \"77044b62-3256-44be-b8d7-245046a12ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt" Nov 04 01:37:19.645875 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:37:19.645781 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:37:19.694439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:19.693344 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt" Nov 04 01:37:20.205709 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.205670 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt"] Nov 04 01:37:20.210579 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:37:20.210546 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77044b62_3256_44be_b8d7_245046a12ff1.slice/crio-f2b1da74b48f8e1fef2705fa8621c955309e25caed9f8fc1921cd8907ad8830d WatchSource:0}: Error finding container f2b1da74b48f8e1fef2705fa8621c955309e25caed9f8fc1921cd8907ad8830d: Status 404 returned error can't find the container with id f2b1da74b48f8e1fef2705fa8621c955309e25caed9f8fc1921cd8907ad8830d Nov 04 01:37:20.774170 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.774095 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp"] Nov 04 01:37:20.777799 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.777779 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" Nov 04 01:37:20.827180 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.827137 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d057d7\"/\"agc-cred-273507bf34c3\"" Nov 04 01:37:20.827180 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.827146 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d057d7\"/\"payload-processing-external-model-e2e--285c2eab-dockercfg-fvxfl\"" Nov 04 01:37:20.827180 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.827152 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d057d7\"/\"openshift-service-ca.crt\"" Nov 04 01:37:20.827180 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.827179 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d057d7\"/\"routing-overlay\"" Nov 04 01:37:20.877077 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.877046 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp"] Nov 04 01:37:20.894806 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.894770 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0686885-ff19-476d-9bff-c5a02d5855ae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp\" (UID: \"f0686885-ff19-476d-9bff-c5a02d5855ae\") " pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" Nov 04 01:37:20.894976 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.894813 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0686885-ff19-476d-9bff-c5a02d5855ae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp\" (UID: \"f0686885-ff19-476d-9bff-c5a02d5855ae\") " pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" Nov 04 01:37:20.894976 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.894942 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0686885-ff19-476d-9bff-c5a02d5855ae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp\" (UID: \"f0686885-ff19-476d-9bff-c5a02d5855ae\") " pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" Nov 04 01:37:20.895049 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.895023 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqfzm\" (UniqueName: \"kubernetes.io/projected/f0686885-ff19-476d-9bff-c5a02d5855ae-kube-api-access-sqfzm\") pod \"payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp\" (UID: \"f0686885-ff19-476d-9bff-c5a02d5855ae\") " pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" Nov 04 01:37:20.928312 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.928283 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d057d7\"/\"kube-root-ca.crt\"" Nov 04 01:37:20.928500 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.928337 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d057d7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d057d7\"" Nov 04 01:37:20.996674 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.996638 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0686885-ff19-476d-9bff-c5a02d5855ae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp\" (UID: \"f0686885-ff19-476d-9bff-c5a02d5855ae\") " pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" Nov 04 01:37:20.996829 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.996694 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqfzm\" (UniqueName: \"kubernetes.io/projected/f0686885-ff19-476d-9bff-c5a02d5855ae-kube-api-access-sqfzm\") pod \"payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp\" (UID: \"f0686885-ff19-476d-9bff-c5a02d5855ae\") " pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" Nov 04 01:37:20.996889 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.996836 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0686885-ff19-476d-9bff-c5a02d5855ae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp\" (UID: \"f0686885-ff19-476d-9bff-c5a02d5855ae\") " pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" Nov 04 01:37:20.996889 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.996865 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0686885-ff19-476d-9bff-c5a02d5855ae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp\" (UID: \"f0686885-ff19-476d-9bff-c5a02d5855ae\") " pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" Nov 04 01:37:20.997704 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.997679 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0686885-ff19-476d-9bff-c5a02d5855ae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp\" (UID: \"f0686885-ff19-476d-9bff-c5a02d5855ae\") " pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" Nov 04 01:37:20.997778 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.997712 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0686885-ff19-476d-9bff-c5a02d5855ae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp\" (UID: \"f0686885-ff19-476d-9bff-c5a02d5855ae\") " pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" Nov 04 01:37:20.999043 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:20.999025 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0686885-ff19-476d-9bff-c5a02d5855ae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp\" (UID: \"f0686885-ff19-476d-9bff-c5a02d5855ae\") " pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" Nov 04 01:37:21.054672 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:21.054577 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqfzm\" (UniqueName: \"kubernetes.io/projected/f0686885-ff19-476d-9bff-c5a02d5855ae-kube-api-access-sqfzm\") pod \"payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp\" (UID: \"f0686885-ff19-476d-9bff-c5a02d5855ae\") " pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" Nov 04 01:37:21.090813 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:21.090787 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" Nov 04 01:37:21.130562 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:21.130530 5995 generic.go:356] "Generic (PLEG): container finished" podID="77044b62-3256-44be-b8d7-245046a12ff1" containerID="7fd25331179f4e89c2cdfdcee711194b8525f25ba60641341004656f262115f4" exitCode=0 Nov 04 01:37:21.130717 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:21.130613 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt" event={"ID":"77044b62-3256-44be-b8d7-245046a12ff1","Type":"ContainerDied","Data":"7fd25331179f4e89c2cdfdcee711194b8525f25ba60641341004656f262115f4"} Nov 04 01:37:21.130717 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:21.130667 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt" event={"ID":"77044b62-3256-44be-b8d7-245046a12ff1","Type":"ContainerStarted","Data":"f2b1da74b48f8e1fef2705fa8621c955309e25caed9f8fc1921cd8907ad8830d"} Nov 04 01:37:21.468732 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:21.468702 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp"] Nov 04 01:37:21.473830 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:37:21.473802 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0686885_ff19_476d_9bff_c5a02d5855ae.slice/crio-0714bac1a3d63608188c76ae7d81e34063d375d612f418d144895c30a5fff22d WatchSource:0}: Error finding container 0714bac1a3d63608188c76ae7d81e34063d375d612f418d144895c30a5fff22d: Status 404 returned error can't find the container with id 0714bac1a3d63608188c76ae7d81e34063d375d612f418d144895c30a5fff22d Nov 04 01:37:22.137690 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:22.137656 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" event={"ID":"f0686885-ff19-476d-9bff-c5a02d5855ae","Type":"ContainerStarted","Data":"4790695f45e48e3f2511663f8d4f2d4fba41bfbf3c3fa0926376af609b976308"} Nov 04 01:37:22.137872 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:22.137699 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" event={"ID":"f0686885-ff19-476d-9bff-c5a02d5855ae","Type":"ContainerStarted","Data":"0714bac1a3d63608188c76ae7d81e34063d375d612f418d144895c30a5fff22d"} Nov 04 01:37:22.137872 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:22.137858 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" Nov 04 01:37:22.253490 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:22.253430 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" podStartSLOduration=2.253393918 podStartE2EDuration="2.253393918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:37:22.248019316 +0000 UTC m=+1592.279752730" watchObservedRunningTime="2026-11-04 01:37:22.253393918 +0000 UTC m=+1592.285127309" Nov 04 01:37:22.292539 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:22.292508 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt" Nov 04 01:37:22.418486 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:22.414942 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77044b62-3256-44be-b8d7-245046a12ff1-kube-api-access-gnc7w\" (UniqueName: \"kubernetes.io/projected/77044b62-3256-44be-b8d7-245046a12ff1-kube-api-access-gnc7w\") pod \"77044b62-3256-44be-b8d7-245046a12ff1\" (UID: \"77044b62-3256-44be-b8d7-245046a12ff1\") " Nov 04 01:37:22.418486 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:22.415102 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77044b62-3256-44be-b8d7-245046a12ff1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77044b62-3256-44be-b8d7-245046a12ff1-maas-api-openshift-ca\") pod \"77044b62-3256-44be-b8d7-245046a12ff1\" (UID: \"77044b62-3256-44be-b8d7-245046a12ff1\") " Nov 04 01:37:22.418486 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:22.415223 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77044b62-3256-44be-b8d7-245046a12ff1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77044b62-3256-44be-b8d7-245046a12ff1-maas-api-xks-ca\") pod \"77044b62-3256-44be-b8d7-245046a12ff1\" (UID: \"77044b62-3256-44be-b8d7-245046a12ff1\") " Nov 04 01:37:22.418486 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:22.416410 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77044b62-3256-44be-b8d7-245046a12ff1-maas-api-openshift-ca" pod "77044b62-3256-44be-b8d7-245046a12ff1" (UID: "77044b62-3256-44be-b8d7-245046a12ff1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:37:22.422617 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:22.419480 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77044b62-3256-44be-b8d7-245046a12ff1-kube-api-access-gnc7w" pod "77044b62-3256-44be-b8d7-245046a12ff1" (UID: "77044b62-3256-44be-b8d7-245046a12ff1"). InnerVolumeSpecName "kube-api-access-gnc7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:37:22.424038 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:22.424004 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77044b62-3256-44be-b8d7-245046a12ff1-maas-api-xks-ca" pod "77044b62-3256-44be-b8d7-245046a12ff1" (UID: "77044b62-3256-44be-b8d7-245046a12ff1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:37:22.517178 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:22.517134 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77044b62-3256-44be-b8d7-245046a12ff1-maas-api-openshift-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:22.517178 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:22.517170 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77044b62-3256-44be-b8d7-245046a12ff1-maas-api-xks-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:22.517178 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:22.517180 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnc7w\" (UniqueName: \"kubernetes.io/projected/77044b62-3256-44be-b8d7-245046a12ff1-kube-api-access-gnc7w\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:23.058511 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.058448 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:37:23.124128 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.124076 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-credential-socket\") pod \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " Nov 04 01:37:23.124276 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.124162 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-token\" (UniqueName: \"kubernetes.io/projected/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-token\") pod \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " Nov 04 01:37:23.124276 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.124242 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-workload-certs\") pod \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " Nov 04 01:37:23.124399 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.124309 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-data\") pod \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " Nov 04 01:37:23.124399 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.124381 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-envoy\") pod \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " Nov 04 01:37:23.124595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.124386 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-credential-socket" pod "cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" (UID: "cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:37:23.124595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.124456 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-workload-socket\") pod \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " Nov 04 01:37:23.124595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.124533 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-podinfo\") pod \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " Nov 04 01:37:23.124595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.124565 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istiod-ca-cert\") pod \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " Nov 04 01:37:23.124797 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.124595 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-data" pod "cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" (UID: "cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:37:23.124797 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.124607 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-workload-certs" pod "cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" (UID: "cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:37:23.124797 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.124635 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-kube-api-access-jj7xw\" (UniqueName: \"kubernetes.io/projected/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-kube-api-access-jj7xw\") pod \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\" (UID: \"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2\") " Nov 04 01:37:23.124797 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.124724 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-workload-socket" pod "cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" (UID: "cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:37:23.125000 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.124947 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istiod-ca-cert" pod "cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" (UID: "cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:37:23.125127 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.125106 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:23.125188 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.125132 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:23.125188 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.125146 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:23.125188 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.125160 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:23.125188 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.125173 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:23.126806 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.126783 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-podinfo" pod "cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" (UID: "cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:37:23.126911 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.126856 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-kube-api-access-jj7xw" pod "cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" (UID: "cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2"). InnerVolumeSpecName "kube-api-access-jj7xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:37:23.126965 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.126937 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-envoy" pod "cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" (UID: "cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:37:23.127014 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.126972 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-token" pod "cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" (UID: "cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:37:23.146164 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.146133 5995 generic.go:356] "Generic (PLEG): container finished" podID="cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" containerID="3727b1294557a5d235d3e7c5430e36331cb9acfc29fa0eaeeb13e39db390d5f3" exitCode=0 Nov 04 01:37:23.146277 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.146194 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" Nov 04 01:37:23.146277 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.146225 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" event={"ID":"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2","Type":"ContainerDied","Data":"3727b1294557a5d235d3e7c5430e36331cb9acfc29fa0eaeeb13e39db390d5f3"} Nov 04 01:37:23.146440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.146294 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6" event={"ID":"cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2","Type":"ContainerDied","Data":"63517e5c8640b9975051393847dcdc89a82bf058c7830b471f4fec5a3100bf9d"} Nov 04 01:37:23.146440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.146325 5995 scope.go:122] "RemoveContainer" containerID="3727b1294557a5d235d3e7c5430e36331cb9acfc29fa0eaeeb13e39db390d5f3" Nov 04 01:37:23.147881 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.147858 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt" event={"ID":"77044b62-3256-44be-b8d7-245046a12ff1","Type":"ContainerDied","Data":"f2b1da74b48f8e1fef2705fa8621c955309e25caed9f8fc1921cd8907ad8830d"} Nov 04 01:37:23.147974 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.147899 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt" Nov 04 01:37:23.147974 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.147906 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2b1da74b48f8e1fef2705fa8621c955309e25caed9f8fc1921cd8907ad8830d" Nov 04 01:37:23.157151 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.157126 5995 scope.go:122] "RemoveContainer" containerID="ec5a5dc4c591c23e8a1a88868d03ed8a7d41a8a797cd6cd220364f5bb701be09" Nov 04 01:37:23.168055 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.168034 5995 scope.go:122] "RemoveContainer" containerID="3727b1294557a5d235d3e7c5430e36331cb9acfc29fa0eaeeb13e39db390d5f3" Nov 04 01:37:23.168340 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:37:23.168323 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3727b1294557a5d235d3e7c5430e36331cb9acfc29fa0eaeeb13e39db390d5f3\": container with ID starting with 3727b1294557a5d235d3e7c5430e36331cb9acfc29fa0eaeeb13e39db390d5f3 not found: ID does not exist" containerID="3727b1294557a5d235d3e7c5430e36331cb9acfc29fa0eaeeb13e39db390d5f3" Nov 04 01:37:23.168407 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.168348 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3727b1294557a5d235d3e7c5430e36331cb9acfc29fa0eaeeb13e39db390d5f3"} err="failed to get container status \"3727b1294557a5d235d3e7c5430e36331cb9acfc29fa0eaeeb13e39db390d5f3\": rpc error: code = NotFound desc = could not find container \"3727b1294557a5d235d3e7c5430e36331cb9acfc29fa0eaeeb13e39db390d5f3\": container with ID starting with 3727b1294557a5d235d3e7c5430e36331cb9acfc29fa0eaeeb13e39db390d5f3 not found: ID does not exist" Nov 04 01:37:23.168407 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.168365 5995 scope.go:122] "RemoveContainer" containerID="ec5a5dc4c591c23e8a1a88868d03ed8a7d41a8a797cd6cd220364f5bb701be09" Nov 04 01:37:23.168629 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:37:23.168612 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec5a5dc4c591c23e8a1a88868d03ed8a7d41a8a797cd6cd220364f5bb701be09\": container with ID starting with ec5a5dc4c591c23e8a1a88868d03ed8a7d41a8a797cd6cd220364f5bb701be09 not found: ID does not exist" containerID="ec5a5dc4c591c23e8a1a88868d03ed8a7d41a8a797cd6cd220364f5bb701be09" Nov 04 01:37:23.168691 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.168634 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5a5dc4c591c23e8a1a88868d03ed8a7d41a8a797cd6cd220364f5bb701be09"} err="failed to get container status \"ec5a5dc4c591c23e8a1a88868d03ed8a7d41a8a797cd6cd220364f5bb701be09\": rpc error: code = NotFound desc = could not find container \"ec5a5dc4c591c23e8a1a88868d03ed8a7d41a8a797cd6cd220364f5bb701be09\": container with ID starting with ec5a5dc4c591c23e8a1a88868d03ed8a7d41a8a797cd6cd220364f5bb701be09 not found: ID does not exist" Nov 04 01:37:23.237451 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.231505 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:23.237451 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.231540 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:23.237451 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.231556 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj7xw\" (UniqueName: \"kubernetes.io/projected/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-kube-api-access-jj7xw\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:23.237451 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.231577 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:23.333782 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.333646 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6"] Nov 04 01:37:23.338308 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:23.338279 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-cae0ab-openshift-default-b65d947d9-tkfd6"] Nov 04 01:37:24.129660 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:24.129628 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" path="/var/lib/kubelet/pods/cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2/volumes" Nov 04 01:37:25.867643 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:37:25.867604 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bec17d7cdfd4: secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:37:25.867643 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:37:25.867644 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz: secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:37:25.868101 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:37:25.867717 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials podName:7650b8ae-725e-470e-8960-5053aa977ea0 nodeName:}" failed. No retries permitted until 2026-11-04 01:38:29.867694459 +0000 UTC m=+1659.899427842 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials") pod "payload-processing-external-model-6fddb87f65-hxqhz" (UID: "7650b8ae-725e-470e-8960-5053aa977ea0") : secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:37:28.069937 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:28.069889 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:37:29.123180 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:29.123145 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:37:30.450992 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.450951 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv"] Nov 04 01:37:30.451636 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.451617 5995 cpu_manager.go:414] "containerMap: removing container" podUID="77044b62-3256-44be-b8d7-245046a12ff1" containerName="revoke-keys" Nov 04 01:37:30.451636 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.451637 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77044b62-3256-44be-b8d7-245046a12ff1" containerName="revoke-keys" Nov 04 01:37:30.451762 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.451656 5995 cpu_manager.go:414] "containerMap: removing container" podUID="cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" containerName="istio-proxy" Nov 04 01:37:30.451762 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.451662 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" containerName="istio-proxy" Nov 04 01:37:30.451762 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.451688 5995 cpu_manager.go:414] "containerMap: removing container" podUID="cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" containerName="istio-proxy" Nov 04 01:37:30.451762 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.451696 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" containerName="istio-proxy" Nov 04 01:37:30.451892 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.451808 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" containerName="istio-proxy" Nov 04 01:37:30.451892 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.451823 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc9f1fa2-1c99-4c07-ab88-aafd1f5d5bb2" containerName="istio-proxy" Nov 04 01:37:30.451892 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.451840 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="77044b62-3256-44be-b8d7-245046a12ff1" containerName="revoke-keys" Nov 04 01:37:30.456603 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.456584 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv" Nov 04 01:37:30.505972 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.505938 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:37:30.554218 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.552594 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:37:30.554218 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.552668 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:37:30.554218 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.552772 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w799x\"" Nov 04 01:37:30.554218 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.553094 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46569cf9-9dc0-4933-a5e1-7a72f5004728-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71e1959d-8f8hv\" (UID: \"46569cf9-9dc0-4933-a5e1-7a72f5004728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv" Nov 04 01:37:30.554218 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.553294 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46569cf9-9dc0-4933-a5e1-7a72f5004728-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71e1959d-8f8hv\" (UID: \"46569cf9-9dc0-4933-a5e1-7a72f5004728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv" Nov 04 01:37:30.554218 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.553332 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6h74\" (UniqueName: \"kubernetes.io/projected/46569cf9-9dc0-4933-a5e1-7a72f5004728-kube-api-access-z6h74\") pod \"maas-api-revoke-sub-71e1959d-8f8hv\" (UID: \"46569cf9-9dc0-4933-a5e1-7a72f5004728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv" Nov 04 01:37:30.555782 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.555758 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv"] Nov 04 01:37:30.654701 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.654664 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46569cf9-9dc0-4933-a5e1-7a72f5004728-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71e1959d-8f8hv\" (UID: \"46569cf9-9dc0-4933-a5e1-7a72f5004728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv" Nov 04 01:37:30.654911 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.654786 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46569cf9-9dc0-4933-a5e1-7a72f5004728-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71e1959d-8f8hv\" (UID: \"46569cf9-9dc0-4933-a5e1-7a72f5004728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv" Nov 04 01:37:30.654911 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.654808 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6h74\" (UniqueName: \"kubernetes.io/projected/46569cf9-9dc0-4933-a5e1-7a72f5004728-kube-api-access-z6h74\") pod \"maas-api-revoke-sub-71e1959d-8f8hv\" (UID: \"46569cf9-9dc0-4933-a5e1-7a72f5004728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv" Nov 04 01:37:30.655552 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.655514 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46569cf9-9dc0-4933-a5e1-7a72f5004728-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71e1959d-8f8hv\" (UID: \"46569cf9-9dc0-4933-a5e1-7a72f5004728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv" Nov 04 01:37:30.657090 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.657059 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46569cf9-9dc0-4933-a5e1-7a72f5004728-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71e1959d-8f8hv\" (UID: \"46569cf9-9dc0-4933-a5e1-7a72f5004728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv" Nov 04 01:37:30.713078 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.713005 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6h74\" (UniqueName: \"kubernetes.io/projected/46569cf9-9dc0-4933-a5e1-7a72f5004728-kube-api-access-z6h74\") pod \"maas-api-revoke-sub-71e1959d-8f8hv\" (UID: \"46569cf9-9dc0-4933-a5e1-7a72f5004728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv" Nov 04 01:37:30.770996 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:30.770967 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv" Nov 04 01:37:31.143072 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:31.142968 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv"] Nov 04 01:37:31.148395 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:37:31.148366 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46569cf9_9dc0_4933_a5e1_7a72f5004728.slice/crio-88b112a690eedff935ac235dfb8fe2274f15ed4c8ef0ed9b4b6958d1a9d1c22d WatchSource:0}: Error finding container 88b112a690eedff935ac235dfb8fe2274f15ed4c8ef0ed9b4b6958d1a9d1c22d: Status 404 returned error can't find the container with id 88b112a690eedff935ac235dfb8fe2274f15ed4c8ef0ed9b4b6958d1a9d1c22d Nov 04 01:37:31.206764 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:31.206733 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv" event={"ID":"46569cf9-9dc0-4933-a5e1-7a72f5004728","Type":"ContainerStarted","Data":"88b112a690eedff935ac235dfb8fe2274f15ed4c8ef0ed9b4b6958d1a9d1c22d"} Nov 04 01:37:32.216021 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:32.215985 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv" event={"ID":"46569cf9-9dc0-4933-a5e1-7a72f5004728","Type":"ContainerStarted","Data":"7d0a61072879a14f7421be093e0f127e7d8693b6defb93daf8a6d664eb9f6868"} Nov 04 01:37:32.328835 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:32.328780 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv" podStartSLOduration=2.328766638 podStartE2EDuration="2.328766638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:37:32.326275287 +0000 UTC m=+1602.358008679" watchObservedRunningTime="2026-11-04 01:37:32.328766638 +0000 UTC m=+1602.360500028" Nov 04 01:37:33.151902 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:33.151873 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" Nov 04 01:37:33.224333 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:33.224297 5995 generic.go:356] "Generic (PLEG): container finished" podID="46569cf9-9dc0-4933-a5e1-7a72f5004728" containerID="7d0a61072879a14f7421be093e0f127e7d8693b6defb93daf8a6d664eb9f6868" exitCode=0 Nov 04 01:37:33.224756 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:33.224337 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv" event={"ID":"46569cf9-9dc0-4933-a5e1-7a72f5004728","Type":"ContainerDied","Data":"7d0a61072879a14f7421be093e0f127e7d8693b6defb93daf8a6d664eb9f6868"} Nov 04 01:37:34.452150 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:34.452119 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv" Nov 04 01:37:34.510145 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:34.510013 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46569cf9-9dc0-4933-a5e1-7a72f5004728-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46569cf9-9dc0-4933-a5e1-7a72f5004728-maas-api-openshift-ca\") pod \"46569cf9-9dc0-4933-a5e1-7a72f5004728\" (UID: \"46569cf9-9dc0-4933-a5e1-7a72f5004728\") " Nov 04 01:37:34.510316 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:34.510161 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46569cf9-9dc0-4933-a5e1-7a72f5004728-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46569cf9-9dc0-4933-a5e1-7a72f5004728-maas-api-xks-ca\") pod \"46569cf9-9dc0-4933-a5e1-7a72f5004728\" (UID: \"46569cf9-9dc0-4933-a5e1-7a72f5004728\") " Nov 04 01:37:34.510316 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:34.510239 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46569cf9-9dc0-4933-a5e1-7a72f5004728-kube-api-access-z6h74\" (UniqueName: \"kubernetes.io/projected/46569cf9-9dc0-4933-a5e1-7a72f5004728-kube-api-access-z6h74\") pod \"46569cf9-9dc0-4933-a5e1-7a72f5004728\" (UID: \"46569cf9-9dc0-4933-a5e1-7a72f5004728\") " Nov 04 01:37:34.510494 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:34.510468 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46569cf9-9dc0-4933-a5e1-7a72f5004728-maas-api-openshift-ca" pod "46569cf9-9dc0-4933-a5e1-7a72f5004728" (UID: "46569cf9-9dc0-4933-a5e1-7a72f5004728"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:37:34.510747 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:34.510730 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46569cf9-9dc0-4933-a5e1-7a72f5004728-maas-api-openshift-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:34.512449 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:34.512423 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46569cf9-9dc0-4933-a5e1-7a72f5004728-kube-api-access-z6h74" pod "46569cf9-9dc0-4933-a5e1-7a72f5004728" (UID: "46569cf9-9dc0-4933-a5e1-7a72f5004728"). InnerVolumeSpecName "kube-api-access-z6h74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:37:34.512554 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:34.512498 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46569cf9-9dc0-4933-a5e1-7a72f5004728-maas-api-xks-ca" pod "46569cf9-9dc0-4933-a5e1-7a72f5004728" (UID: "46569cf9-9dc0-4933-a5e1-7a72f5004728"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:37:34.611889 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:34.611840 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6h74\" (UniqueName: \"kubernetes.io/projected/46569cf9-9dc0-4933-a5e1-7a72f5004728-kube-api-access-z6h74\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:34.611889 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:34.611880 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46569cf9-9dc0-4933-a5e1-7a72f5004728-maas-api-xks-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:35.239528 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:35.239496 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv" Nov 04 01:37:35.239712 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:35.239496 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv" event={"ID":"46569cf9-9dc0-4933-a5e1-7a72f5004728","Type":"ContainerDied","Data":"88b112a690eedff935ac235dfb8fe2274f15ed4c8ef0ed9b4b6958d1a9d1c22d"} Nov 04 01:37:35.239712 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:35.239637 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88b112a690eedff935ac235dfb8fe2274f15ed4c8ef0ed9b4b6958d1a9d1c22d" Nov 04 01:37:39.123193 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:39.123160 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:37:49.123974 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:49.123937 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:37:49.642661 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:37:49.642625 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:37:51.630625 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.630588 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h"] Nov 04 01:37:51.631314 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.631289 5995 cpu_manager.go:414] "containerMap: removing container" podUID="46569cf9-9dc0-4933-a5e1-7a72f5004728" containerName="revoke-keys" Nov 04 01:37:51.631314 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.631314 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46569cf9-9dc0-4933-a5e1-7a72f5004728" containerName="revoke-keys" Nov 04 01:37:51.631515 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.631507 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="46569cf9-9dc0-4933-a5e1-7a72f5004728" containerName="revoke-keys" Nov 04 01:37:51.636452 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.636433 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" Nov 04 01:37:51.686103 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.686071 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1225f3\"/\"kube-root-ca.crt\"" Nov 04 01:37:51.686103 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.686072 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1225f3\"/\"routing-overlay\"" Nov 04 01:37:51.732813 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.732780 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1225f3\"/\"agc-cred-300f65de7d76\"" Nov 04 01:37:51.732813 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.732781 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1225f3\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1225f3\"" Nov 04 01:37:51.733066 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.732913 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1225f3\"/\"payload-processing-external-model-e2e--1d554d3b-dockercfg-v2tjb\"" Nov 04 01:37:51.733066 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.732939 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1225f3\"/\"openshift-service-ca.crt\"" Nov 04 01:37:51.737889 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.737867 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h"] Nov 04 01:37:51.794868 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.794832 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h\" (UID: \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\") " pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" Nov 04 01:37:51.795043 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.794877 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h\" (UID: \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\") " pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" Nov 04 01:37:51.795043 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.795028 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h\" (UID: \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\") " pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" Nov 04 01:37:51.795172 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.795158 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqgf5\" (UniqueName: \"kubernetes.io/projected/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-kube-api-access-dqgf5\") pod \"payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h\" (UID: \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\") " pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" Nov 04 01:37:51.896689 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.896603 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h\" (UID: \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\") " pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" Nov 04 01:37:51.896689 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.896663 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h\" (UID: \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\") " pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" Nov 04 01:37:51.896859 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.896741 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h\" (UID: \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\") " pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" Nov 04 01:37:51.896859 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.896823 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqgf5\" (UniqueName: \"kubernetes.io/projected/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-kube-api-access-dqgf5\") pod \"payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h\" (UID: \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\") " pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" Nov 04 01:37:51.897409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.897387 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h\" (UID: \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\") " pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" Nov 04 01:37:51.897543 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.897481 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h\" (UID: \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\") " pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" Nov 04 01:37:51.899275 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.899250 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h\" (UID: \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\") " pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" Nov 04 01:37:51.955475 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:51.955444 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqgf5\" (UniqueName: \"kubernetes.io/projected/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-kube-api-access-dqgf5\") pod \"payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h\" (UID: \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\") " pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" Nov 04 01:37:52.249886 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:52.249840 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" Nov 04 01:37:52.626580 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:52.626499 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h"] Nov 04 01:37:52.632158 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:37:52.632118 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5fadc3a_fea6_4a2a_b0e6_f197ce2ef629.slice/crio-22e74e9475ba168e084c85913a2faa9fade55c2290370e88726b48c84f1c0a75 WatchSource:0}: Error finding container 22e74e9475ba168e084c85913a2faa9fade55c2290370e88726b48c84f1c0a75: Status 404 returned error can't find the container with id 22e74e9475ba168e084c85913a2faa9fade55c2290370e88726b48c84f1c0a75 Nov 04 01:37:53.391943 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:53.391902 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" event={"ID":"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629","Type":"ContainerStarted","Data":"a9d362002b5f99571eebaf846480a0ccefbddd1db269ae2e97c0d60d7a6b93be"} Nov 04 01:37:53.392123 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:53.391958 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" event={"ID":"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629","Type":"ContainerStarted","Data":"22e74e9475ba168e084c85913a2faa9fade55c2290370e88726b48c84f1c0a75"} Nov 04 01:37:53.392123 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:53.392076 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" Nov 04 01:37:53.509023 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:53.508971 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" podStartSLOduration=2.50895602 podStartE2EDuration="2.50895602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:37:53.505090678 +0000 UTC m=+1623.536824068" watchObservedRunningTime="2026-11-04 01:37:53.50895602 +0000 UTC m=+1623.540689448" Nov 04 01:37:58.623633 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:58.623541 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd"] Nov 04 01:37:58.624169 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:58.623865 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" podUID="94c7bbe4-d255-480e-a031-b9987c89f47b" containerName="istio-proxy" containerID="cri-o://b4c8d845430668ab30a246325fe3bba4f09ada557eae8b020af5febd82a195d1" gracePeriod=30 Nov 04 01:37:58.670370 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:58.670337 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw"] Nov 04 01:37:58.670588 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:58.670558 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" podUID="aca25313-a6c6-4930-920f-488a72e6a3f6" containerName="istio-proxy" containerID="cri-o://87d3b87013c0a55c8bbcaf7907e8c30692cd11bdccc3032b1fecf6c261fb4a9d" gracePeriod=30 Nov 04 01:37:59.123788 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:37:59.123749 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:38:01.176849 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.176811 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74"] Nov 04 01:38:01.181462 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.181442 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.211614 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.211577 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1798ca5e-faea-498a-941a-c810c4c69e33-istio-token\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.211614 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.211616 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-istio-data\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.211831 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.211635 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1798ca5e-faea-498a-941a-c810c4c69e33-istio-podinfo\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.211831 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.211758 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-istio-envoy\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.211831 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.211814 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5xmb\" (UniqueName: \"kubernetes.io/projected/1798ca5e-faea-498a-941a-c810c4c69e33-kube-api-access-j5xmb\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.211925 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.211845 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1798ca5e-faea-498a-941a-c810c4c69e33-istiod-ca-cert\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.211961 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.211939 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-credential-socket\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.211961 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.211957 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-workload-certs\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.212020 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.211987 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-workload-socket\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.230660 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.230624 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9f7183-openshift-default-dockercfg-4jw5f\"" Nov 04 01:38:01.281382 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.281344 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74"] Nov 04 01:38:01.313503 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.313461 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5xmb\" (UniqueName: \"kubernetes.io/projected/1798ca5e-faea-498a-941a-c810c4c69e33-kube-api-access-j5xmb\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.313503 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.313507 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1798ca5e-faea-498a-941a-c810c4c69e33-istiod-ca-cert\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.313926 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.313891 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-credential-socket\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.314087 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.313937 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-workload-certs\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.314087 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.313987 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-workload-socket\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.314208 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.314094 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1798ca5e-faea-498a-941a-c810c4c69e33-istio-token\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.314208 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.314124 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-istio-data\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.314208 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.314136 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1798ca5e-faea-498a-941a-c810c4c69e33-istiod-ca-cert\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.314208 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.314152 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1798ca5e-faea-498a-941a-c810c4c69e33-istio-podinfo\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.314512 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.314210 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-istio-envoy\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.314512 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.314235 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-credential-socket\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.314512 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.314376 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-workload-socket\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.314726 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.314703 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-workload-certs\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.314898 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.314882 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-istio-data\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.317043 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.317017 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-istio-envoy\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.317165 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.317154 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1798ca5e-faea-498a-941a-c810c4c69e33-istio-podinfo\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.371754 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.371724 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1798ca5e-faea-498a-941a-c810c4c69e33-istio-token\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.371940 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.371896 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5xmb\" (UniqueName: \"kubernetes.io/projected/1798ca5e-faea-498a-941a-c810c4c69e33-kube-api-access-j5xmb\") pod \"e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.495015 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.494979 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:01.926805 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.926678 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74"] Nov 04 01:38:01.932180 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:38:01.932149 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1798ca5e_faea_498a_941a_c810c4c69e33.slice/crio-d213dac0b20a6ba463bde58ff607ac7e6a9e31f546121811cff261e66d5d866e WatchSource:0}: Error finding container d213dac0b20a6ba463bde58ff607ac7e6a9e31f546121811cff261e66d5d866e: Status 404 returned error can't find the container with id d213dac0b20a6ba463bde58ff607ac7e6a9e31f546121811cff261e66d5d866e Nov 04 01:38:01.935075 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.935039 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:38:01.935162 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.935107 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:38:01.935162 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:01.935136 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:38:02.458398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:02.458358 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" event={"ID":"1798ca5e-faea-498a-941a-c810c4c69e33","Type":"ContainerStarted","Data":"a3b680b790aa46568d8eac3fcaf842092364fda0eb3586769e5b66b6b11e3f80"} Nov 04 01:38:02.458874 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:02.458440 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" event={"ID":"1798ca5e-faea-498a-941a-c810c4c69e33","Type":"ContainerStarted","Data":"d213dac0b20a6ba463bde58ff607ac7e6a9e31f546121811cff261e66d5d866e"} Nov 04 01:38:02.496102 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:02.496062 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:02.596573 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:02.596380 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" podStartSLOduration=1.596357365 podStartE2EDuration="1.596357365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:38:02.571898656 +0000 UTC m=+1632.603632046" watchObservedRunningTime="2026-11-04 01:38:02.596357365 +0000 UTC m=+1632.628090756" Nov 04 01:38:03.497694 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:03.497645 5995 patch_prober.go:29] interesting pod/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:38:03.498158 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:03.497734 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" podUID="1798ca5e-faea-498a-941a-c810c4c69e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 04 01:38:03.855283 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:03.854946 5995 patch_prober.go:29] interesting pod/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:38:03.855283 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:03.855006 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" podUID="aca25313-a6c6-4930-920f-488a72e6a3f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 04 01:38:04.005666 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.005639 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:38:04.051944 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.051917 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-credential-socket\") pod \"94c7bbe4-d255-480e-a031-b9987c89f47b\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " Nov 04 01:38:04.052068 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.052043 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c7bbe4-d255-480e-a031-b9987c89f47b-kube-api-access-8g6rg\" (UniqueName: \"kubernetes.io/projected/94c7bbe4-d255-480e-a031-b9987c89f47b-kube-api-access-8g6rg\") pod \"94c7bbe4-d255-480e-a031-b9987c89f47b\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " Nov 04 01:38:04.052111 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.052079 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94c7bbe4-d255-480e-a031-b9987c89f47b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c7bbe4-d255-480e-a031-b9987c89f47b-istiod-ca-cert\") pod \"94c7bbe4-d255-480e-a031-b9987c89f47b\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " Nov 04 01:38:04.052147 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.052131 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-data\") pod \"94c7bbe4-d255-480e-a031-b9987c89f47b\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " Nov 04 01:38:04.052184 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.052167 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-envoy\") pod \"94c7bbe4-d255-480e-a031-b9987c89f47b\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " Nov 04 01:38:04.052227 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.052180 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-credential-socket" pod "94c7bbe4-d255-480e-a031-b9987c89f47b" (UID: "94c7bbe4-d255-480e-a031-b9987c89f47b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:38:04.052281 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.052267 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-workload-certs\") pod \"94c7bbe4-d255-480e-a031-b9987c89f47b\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " Nov 04 01:38:04.052374 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.052357 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-workload-socket\") pod \"94c7bbe4-d255-480e-a031-b9987c89f47b\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " Nov 04 01:38:04.052465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.052397 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-token\" (UniqueName: \"kubernetes.io/projected/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-token\") pod \"94c7bbe4-d255-480e-a031-b9987c89f47b\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " Nov 04 01:38:04.052532 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.052466 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-podinfo\") pod \"94c7bbe4-d255-480e-a031-b9987c89f47b\" (UID: \"94c7bbe4-d255-480e-a031-b9987c89f47b\") " Nov 04 01:38:04.052532 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.052466 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-data" pod "94c7bbe4-d255-480e-a031-b9987c89f47b" (UID: "94c7bbe4-d255-480e-a031-b9987c89f47b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:38:04.052631 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.052563 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c7bbe4-d255-480e-a031-b9987c89f47b-istiod-ca-cert" pod "94c7bbe4-d255-480e-a031-b9987c89f47b" (UID: "94c7bbe4-d255-480e-a031-b9987c89f47b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:38:04.052675 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.052657 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-workload-socket" pod "94c7bbe4-d255-480e-a031-b9987c89f47b" (UID: "94c7bbe4-d255-480e-a031-b9987c89f47b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:38:04.053118 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.053013 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c7bbe4-d255-480e-a031-b9987c89f47b-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:04.053118 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.053043 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:04.053118 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.053057 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:04.053118 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.053072 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:04.053613 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.053586 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-workload-certs" pod "94c7bbe4-d255-480e-a031-b9987c89f47b" (UID: "94c7bbe4-d255-480e-a031-b9987c89f47b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:38:04.054544 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.054522 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c7bbe4-d255-480e-a031-b9987c89f47b-kube-api-access-8g6rg" pod "94c7bbe4-d255-480e-a031-b9987c89f47b" (UID: "94c7bbe4-d255-480e-a031-b9987c89f47b"). InnerVolumeSpecName "kube-api-access-8g6rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:04.054649 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.054630 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-token" pod "94c7bbe4-d255-480e-a031-b9987c89f47b" (UID: "94c7bbe4-d255-480e-a031-b9987c89f47b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:04.054766 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.054747 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-podinfo" pod "94c7bbe4-d255-480e-a031-b9987c89f47b" (UID: "94c7bbe4-d255-480e-a031-b9987c89f47b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:38:04.054815 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.054757 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-envoy" pod "94c7bbe4-d255-480e-a031-b9987c89f47b" (UID: "94c7bbe4-d255-480e-a031-b9987c89f47b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:38:04.060831 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.060813 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:38:04.153994 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.153959 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-credential-socket\") pod \"aca25313-a6c6-4930-920f-488a72e6a3f6\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " Nov 04 01:38:04.154158 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.154009 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aca25313-a6c6-4930-920f-488a72e6a3f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca25313-a6c6-4930-920f-488a72e6a3f6-istiod-ca-cert\") pod \"aca25313-a6c6-4930-920f-488a72e6a3f6\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " Nov 04 01:38:04.154158 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.154123 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-podinfo\") pod \"aca25313-a6c6-4930-920f-488a72e6a3f6\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " Nov 04 01:38:04.154269 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.154208 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-envoy\") pod \"aca25313-a6c6-4930-920f-488a72e6a3f6\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " Nov 04 01:38:04.154269 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.154216 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-credential-socket" pod "aca25313-a6c6-4930-920f-488a72e6a3f6" (UID: "aca25313-a6c6-4930-920f-488a72e6a3f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:38:04.154387 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.154303 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-workload-certs\") pod \"aca25313-a6c6-4930-920f-488a72e6a3f6\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " Nov 04 01:38:04.154387 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.154366 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-workload-socket\") pod \"aca25313-a6c6-4930-920f-488a72e6a3f6\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " Nov 04 01:38:04.154524 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.154386 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca25313-a6c6-4930-920f-488a72e6a3f6-istiod-ca-cert" pod "aca25313-a6c6-4930-920f-488a72e6a3f6" (UID: "aca25313-a6c6-4930-920f-488a72e6a3f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:38:04.154524 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.154482 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-data\") pod \"aca25313-a6c6-4930-920f-488a72e6a3f6\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " Nov 04 01:38:04.154623 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.154566 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca25313-a6c6-4930-920f-488a72e6a3f6-kube-api-access-tk2fg\" (UniqueName: \"kubernetes.io/projected/aca25313-a6c6-4930-920f-488a72e6a3f6-kube-api-access-tk2fg\") pod \"aca25313-a6c6-4930-920f-488a72e6a3f6\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " Nov 04 01:38:04.154623 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.154575 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-workload-certs" pod "aca25313-a6c6-4930-920f-488a72e6a3f6" (UID: "aca25313-a6c6-4930-920f-488a72e6a3f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:38:04.154623 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.154606 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-token\" (UniqueName: \"kubernetes.io/projected/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-token\") pod \"aca25313-a6c6-4930-920f-488a72e6a3f6\" (UID: \"aca25313-a6c6-4930-920f-488a72e6a3f6\") " Nov 04 01:38:04.154757 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.154665 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-workload-socket" pod "aca25313-a6c6-4930-920f-488a72e6a3f6" (UID: "aca25313-a6c6-4930-920f-488a72e6a3f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:38:04.154757 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.154736 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-data" pod "aca25313-a6c6-4930-920f-488a72e6a3f6" (UID: "aca25313-a6c6-4930-920f-488a72e6a3f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:38:04.155064 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.155048 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g6rg\" (UniqueName: \"kubernetes.io/projected/94c7bbe4-d255-480e-a031-b9987c89f47b-kube-api-access-8g6rg\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:04.155149 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.155068 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:04.155149 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.155081 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:04.155149 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.155095 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:04.155149 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.155107 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:04.155149 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.155120 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c7bbe4-d255-480e-a031-b9987c89f47b-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:04.155149 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.155134 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:04.155149 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.155145 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:04.155465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.155158 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca25313-a6c6-4930-920f-488a72e6a3f6-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:04.155465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.155172 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c7bbe4-d255-480e-a031-b9987c89f47b-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:04.197084 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.193875 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-podinfo" pod "aca25313-a6c6-4930-920f-488a72e6a3f6" (UID: "aca25313-a6c6-4930-920f-488a72e6a3f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:38:04.202912 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.202853 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-envoy" pod "aca25313-a6c6-4930-920f-488a72e6a3f6" (UID: "aca25313-a6c6-4930-920f-488a72e6a3f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:38:04.202912 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.202873 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca25313-a6c6-4930-920f-488a72e6a3f6-kube-api-access-tk2fg" pod "aca25313-a6c6-4930-920f-488a72e6a3f6" (UID: "aca25313-a6c6-4930-920f-488a72e6a3f6"). InnerVolumeSpecName "kube-api-access-tk2fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:04.202912 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.202893 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-token" pod "aca25313-a6c6-4930-920f-488a72e6a3f6" (UID: "aca25313-a6c6-4930-920f-488a72e6a3f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:04.256826 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.256795 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:04.256826 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.256823 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:04.256826 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.256832 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk2fg\" (UniqueName: \"kubernetes.io/projected/aca25313-a6c6-4930-920f-488a72e6a3f6-kube-api-access-tk2fg\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:04.257059 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.256843 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca25313-a6c6-4930-920f-488a72e6a3f6-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:04.400909 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.400829 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" Nov 04 01:38:04.495359 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.495323 5995 patch_prober.go:29] interesting pod/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:38:04.495548 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.495376 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" podUID="1798ca5e-faea-498a-941a-c810c4c69e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 04 01:38:04.501642 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.501610 5995 generic.go:356] "Generic (PLEG): container finished" podID="aca25313-a6c6-4930-920f-488a72e6a3f6" containerID="87d3b87013c0a55c8bbcaf7907e8c30692cd11bdccc3032b1fecf6c261fb4a9d" exitCode=0 Nov 04 01:38:04.502092 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.501672 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" Nov 04 01:38:04.502092 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.501697 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" event={"ID":"aca25313-a6c6-4930-920f-488a72e6a3f6","Type":"ContainerDied","Data":"87d3b87013c0a55c8bbcaf7907e8c30692cd11bdccc3032b1fecf6c261fb4a9d"} Nov 04 01:38:04.502092 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.501747 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw" event={"ID":"aca25313-a6c6-4930-920f-488a72e6a3f6","Type":"ContainerDied","Data":"c9be9f2c798d984569aa248064c25bc1fb646657ab47e4c4332be381a07ac426"} Nov 04 01:38:04.502092 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.501766 5995 scope.go:122] "RemoveContainer" containerID="87d3b87013c0a55c8bbcaf7907e8c30692cd11bdccc3032b1fecf6c261fb4a9d" Nov 04 01:38:04.503286 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.503262 5995 generic.go:356] "Generic (PLEG): container finished" podID="94c7bbe4-d255-480e-a031-b9987c89f47b" containerID="b4c8d845430668ab30a246325fe3bba4f09ada557eae8b020af5febd82a195d1" exitCode=0 Nov 04 01:38:04.503531 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.503298 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" Nov 04 01:38:04.503531 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.503345 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" event={"ID":"94c7bbe4-d255-480e-a031-b9987c89f47b","Type":"ContainerDied","Data":"b4c8d845430668ab30a246325fe3bba4f09ada557eae8b020af5febd82a195d1"} Nov 04 01:38:04.503531 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.503385 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd" event={"ID":"94c7bbe4-d255-480e-a031-b9987c89f47b","Type":"ContainerDied","Data":"53b8a554190696fee321f6340b3e1939e61df9d718fc401f9008058dd58d5e46"} Nov 04 01:38:04.513385 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.513367 5995 scope.go:122] "RemoveContainer" containerID="87d3b87013c0a55c8bbcaf7907e8c30692cd11bdccc3032b1fecf6c261fb4a9d" Nov 04 01:38:04.513751 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:38:04.513723 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d3b87013c0a55c8bbcaf7907e8c30692cd11bdccc3032b1fecf6c261fb4a9d\": container with ID starting with 87d3b87013c0a55c8bbcaf7907e8c30692cd11bdccc3032b1fecf6c261fb4a9d not found: ID does not exist" containerID="87d3b87013c0a55c8bbcaf7907e8c30692cd11bdccc3032b1fecf6c261fb4a9d" Nov 04 01:38:04.513813 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.513761 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d3b87013c0a55c8bbcaf7907e8c30692cd11bdccc3032b1fecf6c261fb4a9d"} err="failed to get container status \"87d3b87013c0a55c8bbcaf7907e8c30692cd11bdccc3032b1fecf6c261fb4a9d\": rpc error: code = NotFound desc = could not find container \"87d3b87013c0a55c8bbcaf7907e8c30692cd11bdccc3032b1fecf6c261fb4a9d\": container with ID starting with 87d3b87013c0a55c8bbcaf7907e8c30692cd11bdccc3032b1fecf6c261fb4a9d not found: ID does not exist" Nov 04 01:38:04.513813 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.513784 5995 scope.go:122] "RemoveContainer" containerID="b4c8d845430668ab30a246325fe3bba4f09ada557eae8b020af5febd82a195d1" Nov 04 01:38:04.525640 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.525620 5995 scope.go:122] "RemoveContainer" containerID="b4c8d845430668ab30a246325fe3bba4f09ada557eae8b020af5febd82a195d1" Nov 04 01:38:04.526058 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:38:04.526037 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c8d845430668ab30a246325fe3bba4f09ada557eae8b020af5febd82a195d1\": container with ID starting with b4c8d845430668ab30a246325fe3bba4f09ada557eae8b020af5febd82a195d1 not found: ID does not exist" containerID="b4c8d845430668ab30a246325fe3bba4f09ada557eae8b020af5febd82a195d1" Nov 04 01:38:04.526142 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.526066 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c8d845430668ab30a246325fe3bba4f09ada557eae8b020af5febd82a195d1"} err="failed to get container status \"b4c8d845430668ab30a246325fe3bba4f09ada557eae8b020af5febd82a195d1\": rpc error: code = NotFound desc = could not find container \"b4c8d845430668ab30a246325fe3bba4f09ada557eae8b020af5febd82a195d1\": container with ID starting with b4c8d845430668ab30a246325fe3bba4f09ada557eae8b020af5febd82a195d1 not found: ID does not exist" Nov 04 01:38:04.618723 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.618683 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd"] Nov 04 01:38:04.633161 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.633133 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-sk6xd"] Nov 04 01:38:04.790707 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.790671 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw"] Nov 04 01:38:04.794780 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:04.794756 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00cfe6-openshift-default-64579789c7-gsxkw"] Nov 04 01:38:05.496123 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:05.496083 5995 patch_prober.go:29] interesting pod/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:38:05.496316 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:05.496145 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" podUID="1798ca5e-faea-498a-941a-c810c4c69e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 04 01:38:06.130369 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:06.130328 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c7bbe4-d255-480e-a031-b9987c89f47b" path="/var/lib/kubelet/pods/94c7bbe4-d255-480e-a031-b9987c89f47b/volumes" Nov 04 01:38:06.131017 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:06.130992 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca25313-a6c6-4930-920f-488a72e6a3f6" path="/var/lib/kubelet/pods/aca25313-a6c6-4930-920f-488a72e6a3f6/volumes" Nov 04 01:38:06.496019 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:06.495970 5995 patch_prober.go:29] interesting pod/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:38:06.496209 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:06.496040 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" podUID="1798ca5e-faea-498a-941a-c810c4c69e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 04 01:38:07.496109 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:07.496072 5995 patch_prober.go:29] interesting pod/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:38:07.496606 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:07.496131 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" podUID="1798ca5e-faea-498a-941a-c810c4c69e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 04 01:38:08.678132 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:08.678097 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:08.678829 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:08.678803 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:08.679753 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:08.679731 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:09.123473 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:09.123436 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:38:10.708273 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:10.708242 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:38:16.384978 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.384939 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2"] Nov 04 01:38:16.385739 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.385719 5995 cpu_manager.go:414] "containerMap: removing container" podUID="94c7bbe4-d255-480e-a031-b9987c89f47b" containerName="istio-proxy" Nov 04 01:38:16.385805 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.385740 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c7bbe4-d255-480e-a031-b9987c89f47b" containerName="istio-proxy" Nov 04 01:38:16.385805 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.385771 5995 cpu_manager.go:414] "containerMap: removing container" podUID="aca25313-a6c6-4930-920f-488a72e6a3f6" containerName="istio-proxy" Nov 04 01:38:16.385805 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.385776 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca25313-a6c6-4930-920f-488a72e6a3f6" containerName="istio-proxy" Nov 04 01:38:16.385901 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.385856 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c7bbe4-d255-480e-a031-b9987c89f47b" containerName="istio-proxy" Nov 04 01:38:16.385901 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.385868 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca25313-a6c6-4930-920f-488a72e6a3f6" containerName="istio-proxy" Nov 04 01:38:16.389579 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.389563 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2" Nov 04 01:38:16.440713 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.440678 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:38:16.444306 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.444276 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2"] Nov 04 01:38:16.487285 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.487254 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:38:16.487523 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.487316 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:38:16.487523 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.487362 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w799x\"" Nov 04 01:38:16.550972 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.550932 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31857d37-58de-4a1b-a3bf-856b7f20c244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e01ccdf-rchd2\" (UID: \"31857d37-58de-4a1b-a3bf-856b7f20c244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2" Nov 04 01:38:16.551143 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.550977 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxh7z\" (UniqueName: \"kubernetes.io/projected/31857d37-58de-4a1b-a3bf-856b7f20c244-kube-api-access-fxh7z\") pod \"maas-api-revoke-sub-4e01ccdf-rchd2\" (UID: \"31857d37-58de-4a1b-a3bf-856b7f20c244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2" Nov 04 01:38:16.551143 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.551070 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31857d37-58de-4a1b-a3bf-856b7f20c244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e01ccdf-rchd2\" (UID: \"31857d37-58de-4a1b-a3bf-856b7f20c244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2" Nov 04 01:38:16.652407 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.652331 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31857d37-58de-4a1b-a3bf-856b7f20c244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e01ccdf-rchd2\" (UID: \"31857d37-58de-4a1b-a3bf-856b7f20c244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2" Nov 04 01:38:16.652407 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.652365 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxh7z\" (UniqueName: \"kubernetes.io/projected/31857d37-58de-4a1b-a3bf-856b7f20c244-kube-api-access-fxh7z\") pod \"maas-api-revoke-sub-4e01ccdf-rchd2\" (UID: \"31857d37-58de-4a1b-a3bf-856b7f20c244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2" Nov 04 01:38:16.652624 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.652490 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31857d37-58de-4a1b-a3bf-856b7f20c244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e01ccdf-rchd2\" (UID: \"31857d37-58de-4a1b-a3bf-856b7f20c244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2" Nov 04 01:38:16.653087 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.653068 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31857d37-58de-4a1b-a3bf-856b7f20c244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e01ccdf-rchd2\" (UID: \"31857d37-58de-4a1b-a3bf-856b7f20c244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2" Nov 04 01:38:16.654953 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.654928 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31857d37-58de-4a1b-a3bf-856b7f20c244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e01ccdf-rchd2\" (UID: \"31857d37-58de-4a1b-a3bf-856b7f20c244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2" Nov 04 01:38:16.710320 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:16.710285 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxh7z\" (UniqueName: \"kubernetes.io/projected/31857d37-58de-4a1b-a3bf-856b7f20c244-kube-api-access-fxh7z\") pod \"maas-api-revoke-sub-4e01ccdf-rchd2\" (UID: \"31857d37-58de-4a1b-a3bf-856b7f20c244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2" Nov 04 01:38:17.004050 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:17.004015 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2" Nov 04 01:38:17.376104 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:17.375989 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2"] Nov 04 01:38:17.380901 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:38:17.380875 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31857d37_58de_4a1b_a3bf_856b7f20c244.slice/crio-d16aa02e9368c65aabb7d96223c2090f84028827af91d5acfa55b6b536e7b44c WatchSource:0}: Error finding container d16aa02e9368c65aabb7d96223c2090f84028827af91d5acfa55b6b536e7b44c: Status 404 returned error can't find the container with id d16aa02e9368c65aabb7d96223c2090f84028827af91d5acfa55b6b536e7b44c Nov 04 01:38:17.596661 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:17.596625 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2" event={"ID":"31857d37-58de-4a1b-a3bf-856b7f20c244","Type":"ContainerStarted","Data":"d16aa02e9368c65aabb7d96223c2090f84028827af91d5acfa55b6b536e7b44c"} Nov 04 01:38:18.604762 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:18.604672 5995 generic.go:356] "Generic (PLEG): container finished" podID="31857d37-58de-4a1b-a3bf-856b7f20c244" containerID="3b90d2464e234c608f881f2f0caf5cbaa99c64d9df559e382cc4819461ec6d36" exitCode=0 Nov 04 01:38:18.604762 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:18.604722 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2" event={"ID":"31857d37-58de-4a1b-a3bf-856b7f20c244","Type":"ContainerDied","Data":"3b90d2464e234c608f881f2f0caf5cbaa99c64d9df559e382cc4819461ec6d36"} Nov 04 01:38:19.123060 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:19.123019 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:38:19.648064 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:38:19.648020 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:38:19.771172 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:19.771146 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2" Nov 04 01:38:19.887016 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:19.886927 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31857d37-58de-4a1b-a3bf-856b7f20c244-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31857d37-58de-4a1b-a3bf-856b7f20c244-maas-api-openshift-ca\") pod \"31857d37-58de-4a1b-a3bf-856b7f20c244\" (UID: \"31857d37-58de-4a1b-a3bf-856b7f20c244\") " Nov 04 01:38:19.887016 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:19.886986 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31857d37-58de-4a1b-a3bf-856b7f20c244-kube-api-access-fxh7z\" (UniqueName: \"kubernetes.io/projected/31857d37-58de-4a1b-a3bf-856b7f20c244-kube-api-access-fxh7z\") pod \"31857d37-58de-4a1b-a3bf-856b7f20c244\" (UID: \"31857d37-58de-4a1b-a3bf-856b7f20c244\") " Nov 04 01:38:19.887255 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:19.887102 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31857d37-58de-4a1b-a3bf-856b7f20c244-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31857d37-58de-4a1b-a3bf-856b7f20c244-maas-api-xks-ca\") pod \"31857d37-58de-4a1b-a3bf-856b7f20c244\" (UID: \"31857d37-58de-4a1b-a3bf-856b7f20c244\") " Nov 04 01:38:19.887386 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:19.887361 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31857d37-58de-4a1b-a3bf-856b7f20c244-maas-api-openshift-ca" pod "31857d37-58de-4a1b-a3bf-856b7f20c244" (UID: "31857d37-58de-4a1b-a3bf-856b7f20c244"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:38:19.887644 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:19.887626 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31857d37-58de-4a1b-a3bf-856b7f20c244-maas-api-openshift-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:19.889792 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:19.889759 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31857d37-58de-4a1b-a3bf-856b7f20c244-maas-api-xks-ca" pod "31857d37-58de-4a1b-a3bf-856b7f20c244" (UID: "31857d37-58de-4a1b-a3bf-856b7f20c244"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:38:19.890025 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:19.890005 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31857d37-58de-4a1b-a3bf-856b7f20c244-kube-api-access-fxh7z" pod "31857d37-58de-4a1b-a3bf-856b7f20c244" (UID: "31857d37-58de-4a1b-a3bf-856b7f20c244"). InnerVolumeSpecName "kube-api-access-fxh7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:19.988324 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:19.988291 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxh7z\" (UniqueName: \"kubernetes.io/projected/31857d37-58de-4a1b-a3bf-856b7f20c244-kube-api-access-fxh7z\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:19.988324 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:19.988322 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31857d37-58de-4a1b-a3bf-856b7f20c244-maas-api-xks-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:20.619487 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:20.619448 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2" event={"ID":"31857d37-58de-4a1b-a3bf-856b7f20c244","Type":"ContainerDied","Data":"d16aa02e9368c65aabb7d96223c2090f84028827af91d5acfa55b6b536e7b44c"} Nov 04 01:38:20.619664 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:20.619501 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d16aa02e9368c65aabb7d96223c2090f84028827af91d5acfa55b6b536e7b44c" Nov 04 01:38:20.619664 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:20.619461 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2" Nov 04 01:38:26.444429 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:26.444384 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74"] Nov 04 01:38:26.444855 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:26.444618 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" podUID="1798ca5e-faea-498a-941a-c810c4c69e33" containerName="istio-proxy" containerID="cri-o://a3b680b790aa46568d8eac3fcaf842092364fda0eb3586769e5b66b6b11e3f80" gracePeriod=30 Nov 04 01:38:29.123399 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:29.123367 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:38:29.853436 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:29.833066 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp"] Nov 04 01:38:29.948657 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:38:29.948627 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bec17d7cdfd4: secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:38:29.948657 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:38:29.948656 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz: secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:38:29.948863 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:38:29.948706 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials podName:7650b8ae-725e-470e-8960-5053aa977ea0 nodeName:}" failed. No retries permitted until 2026-11-04 01:40:31.948692832 +0000 UTC m=+1781.980426213 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials") pod "payload-processing-external-model-6fddb87f65-hxqhz" (UID: "7650b8ae-725e-470e-8960-5053aa977ea0") : secret "agc-cred-bec17d7cdfd4" not found Nov 04 01:38:31.711748 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.711671 5995 generic.go:356] "Generic (PLEG): container finished" podID="1798ca5e-faea-498a-941a-c810c4c69e33" containerID="a3b680b790aa46568d8eac3fcaf842092364fda0eb3586769e5b66b6b11e3f80" exitCode=0 Nov 04 01:38:31.711748 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.711709 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" event={"ID":"1798ca5e-faea-498a-941a-c810c4c69e33","Type":"ContainerDied","Data":"a3b680b790aa46568d8eac3fcaf842092364fda0eb3586769e5b66b6b11e3f80"} Nov 04 01:38:31.712175 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.711805 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" event={"ID":"1798ca5e-faea-498a-941a-c810c4c69e33","Type":"ContainerDied","Data":"d213dac0b20a6ba463bde58ff607ac7e6a9e31f546121811cff261e66d5d866e"} Nov 04 01:38:31.712175 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.711829 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d213dac0b20a6ba463bde58ff607ac7e6a9e31f546121811cff261e66d5d866e" Nov 04 01:38:31.715274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.715253 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:31.782308 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.782267 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-istio-data\") pod \"1798ca5e-faea-498a-941a-c810c4c69e33\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " Nov 04 01:38:31.782595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.782318 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1798ca5e-faea-498a-941a-c810c4c69e33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1798ca5e-faea-498a-941a-c810c4c69e33-istiod-ca-cert\") pod \"1798ca5e-faea-498a-941a-c810c4c69e33\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " Nov 04 01:38:31.782595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.782408 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-credential-socket\") pod \"1798ca5e-faea-498a-941a-c810c4c69e33\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " Nov 04 01:38:31.782595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.782471 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1798ca5e-faea-498a-941a-c810c4c69e33-istio-token\" (UniqueName: \"kubernetes.io/projected/1798ca5e-faea-498a-941a-c810c4c69e33-istio-token\") pod \"1798ca5e-faea-498a-941a-c810c4c69e33\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " Nov 04 01:38:31.782595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.782553 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-istio-envoy\") pod \"1798ca5e-faea-498a-941a-c810c4c69e33\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " Nov 04 01:38:31.782595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.782587 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1798ca5e-faea-498a-941a-c810c4c69e33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1798ca5e-faea-498a-941a-c810c4c69e33-istio-podinfo\") pod \"1798ca5e-faea-498a-941a-c810c4c69e33\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " Nov 04 01:38:31.782880 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.782594 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-istio-data" pod "1798ca5e-faea-498a-941a-c810c4c69e33" (UID: "1798ca5e-faea-498a-941a-c810c4c69e33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:38:31.782880 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.782683 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-workload-certs\") pod \"1798ca5e-faea-498a-941a-c810c4c69e33\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " Nov 04 01:38:31.782880 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.782713 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1798ca5e-faea-498a-941a-c810c4c69e33-kube-api-access-j5xmb\" (UniqueName: \"kubernetes.io/projected/1798ca5e-faea-498a-941a-c810c4c69e33-kube-api-access-j5xmb\") pod \"1798ca5e-faea-498a-941a-c810c4c69e33\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " Nov 04 01:38:31.782880 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.782744 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-workload-socket\") pod \"1798ca5e-faea-498a-941a-c810c4c69e33\" (UID: \"1798ca5e-faea-498a-941a-c810c4c69e33\") " Nov 04 01:38:31.782880 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.782765 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1798ca5e-faea-498a-941a-c810c4c69e33-istiod-ca-cert" pod "1798ca5e-faea-498a-941a-c810c4c69e33" (UID: "1798ca5e-faea-498a-941a-c810c4c69e33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:38:31.783248 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.783226 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:31.783341 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.783254 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1798ca5e-faea-498a-941a-c810c4c69e33-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:31.783640 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.783538 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-credential-socket" pod "1798ca5e-faea-498a-941a-c810c4c69e33" (UID: "1798ca5e-faea-498a-941a-c810c4c69e33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:38:31.783927 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.783899 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-workload-certs" pod "1798ca5e-faea-498a-941a-c810c4c69e33" (UID: "1798ca5e-faea-498a-941a-c810c4c69e33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:38:31.785248 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.785223 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-workload-socket" pod "1798ca5e-faea-498a-941a-c810c4c69e33" (UID: "1798ca5e-faea-498a-941a-c810c4c69e33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:38:31.785328 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.785240 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1798ca5e-faea-498a-941a-c810c4c69e33-istio-podinfo" pod "1798ca5e-faea-498a-941a-c810c4c69e33" (UID: "1798ca5e-faea-498a-941a-c810c4c69e33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:38:31.785385 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.785338 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1798ca5e-faea-498a-941a-c810c4c69e33-istio-token" pod "1798ca5e-faea-498a-941a-c810c4c69e33" (UID: "1798ca5e-faea-498a-941a-c810c4c69e33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:31.786904 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.786875 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1798ca5e-faea-498a-941a-c810c4c69e33-kube-api-access-j5xmb" pod "1798ca5e-faea-498a-941a-c810c4c69e33" (UID: "1798ca5e-faea-498a-941a-c810c4c69e33"). InnerVolumeSpecName "kube-api-access-j5xmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:31.786986 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.786929 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-istio-envoy" pod "1798ca5e-faea-498a-941a-c810c4c69e33" (UID: "1798ca5e-faea-498a-941a-c810c4c69e33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:38:31.884733 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.884700 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:31.884733 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.884730 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1798ca5e-faea-498a-941a-c810c4c69e33-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:31.884941 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.884742 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:31.884941 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.884755 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5xmb\" (UniqueName: \"kubernetes.io/projected/1798ca5e-faea-498a-941a-c810c4c69e33-kube-api-access-j5xmb\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:31.884941 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.884768 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:31.884941 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.884782 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1798ca5e-faea-498a-941a-c810c4c69e33-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:31.884941 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:31.884795 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1798ca5e-faea-498a-941a-c810c4c69e33-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:32.720220 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:32.720183 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74" Nov 04 01:38:32.923773 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:32.923734 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74"] Nov 04 01:38:32.969891 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:32.969849 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f7183-openshift-default-646bd94655-dvd74"] Nov 04 01:38:34.123660 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:34.123628 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:38:34.129948 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:34.129922 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1798ca5e-faea-498a-941a-c810c4c69e33" path="/var/lib/kubelet/pods/1798ca5e-faea-498a-941a-c810c4c69e33/volumes" Nov 04 01:38:34.888510 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:34.888454 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" podUID="f0686885-ff19-476d-9bff-c5a02d5855ae" containerName="payload-processing" containerID="cri-o://4790695f45e48e3f2511663f8d4f2d4fba41bfbf3c3fa0926376af609b976308" gracePeriod=25 Nov 04 01:38:35.154224 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.154163 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" Nov 04 01:38:35.230858 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.230824 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0686885-ff19-476d-9bff-c5a02d5855ae-kube-api-access-sqfzm\" (UniqueName: \"kubernetes.io/projected/f0686885-ff19-476d-9bff-c5a02d5855ae-kube-api-access-sqfzm\") pod \"f0686885-ff19-476d-9bff-c5a02d5855ae\" (UID: \"f0686885-ff19-476d-9bff-c5a02d5855ae\") " Nov 04 01:38:35.231038 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.230867 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0686885-ff19-476d-9bff-c5a02d5855ae-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0686885-ff19-476d-9bff-c5a02d5855ae-provider-credentials\") pod \"f0686885-ff19-476d-9bff-c5a02d5855ae\" (UID: \"f0686885-ff19-476d-9bff-c5a02d5855ae\") " Nov 04 01:38:35.231038 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.230926 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0686885-ff19-476d-9bff-c5a02d5855ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0686885-ff19-476d-9bff-c5a02d5855ae-plugins-config-volume\") pod \"f0686885-ff19-476d-9bff-c5a02d5855ae\" (UID: \"f0686885-ff19-476d-9bff-c5a02d5855ae\") " Nov 04 01:38:35.231038 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.230973 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0686885-ff19-476d-9bff-c5a02d5855ae-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0686885-ff19-476d-9bff-c5a02d5855ae-routing-overlay-volume\") pod \"f0686885-ff19-476d-9bff-c5a02d5855ae\" (UID: \"f0686885-ff19-476d-9bff-c5a02d5855ae\") " Nov 04 01:38:35.231385 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.231356 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0686885-ff19-476d-9bff-c5a02d5855ae-plugins-config-volume" pod "f0686885-ff19-476d-9bff-c5a02d5855ae" (UID: "f0686885-ff19-476d-9bff-c5a02d5855ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:38:35.231385 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.231364 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0686885-ff19-476d-9bff-c5a02d5855ae-routing-overlay-volume" pod "f0686885-ff19-476d-9bff-c5a02d5855ae" (UID: "f0686885-ff19-476d-9bff-c5a02d5855ae"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:38:35.233183 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.233160 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0686885-ff19-476d-9bff-c5a02d5855ae-kube-api-access-sqfzm" pod "f0686885-ff19-476d-9bff-c5a02d5855ae" (UID: "f0686885-ff19-476d-9bff-c5a02d5855ae"). InnerVolumeSpecName "kube-api-access-sqfzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:35.233183 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.233173 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0686885-ff19-476d-9bff-c5a02d5855ae-provider-credentials" pod "f0686885-ff19-476d-9bff-c5a02d5855ae" (UID: "f0686885-ff19-476d-9bff-c5a02d5855ae"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:35.332089 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.332057 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqfzm\" (UniqueName: \"kubernetes.io/projected/f0686885-ff19-476d-9bff-c5a02d5855ae-kube-api-access-sqfzm\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:35.332089 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.332086 5995 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0686885-ff19-476d-9bff-c5a02d5855ae-provider-credentials\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:35.332089 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.332096 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0686885-ff19-476d-9bff-c5a02d5855ae-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:35.332321 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.332104 5995 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0686885-ff19-476d-9bff-c5a02d5855ae-routing-overlay-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:35.742207 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.742174 5995 generic.go:356] "Generic (PLEG): container finished" podID="f0686885-ff19-476d-9bff-c5a02d5855ae" containerID="4790695f45e48e3f2511663f8d4f2d4fba41bfbf3c3fa0926376af609b976308" exitCode=0 Nov 04 01:38:35.742401 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.742220 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" Nov 04 01:38:35.742401 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.742258 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" event={"ID":"f0686885-ff19-476d-9bff-c5a02d5855ae","Type":"ContainerDied","Data":"4790695f45e48e3f2511663f8d4f2d4fba41bfbf3c3fa0926376af609b976308"} Nov 04 01:38:35.742401 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.742335 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp" event={"ID":"f0686885-ff19-476d-9bff-c5a02d5855ae","Type":"ContainerDied","Data":"0714bac1a3d63608188c76ae7d81e34063d375d612f418d144895c30a5fff22d"} Nov 04 01:38:35.742401 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.742363 5995 scope.go:122] "RemoveContainer" containerID="4790695f45e48e3f2511663f8d4f2d4fba41bfbf3c3fa0926376af609b976308" Nov 04 01:38:35.757323 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.757298 5995 scope.go:122] "RemoveContainer" containerID="4790695f45e48e3f2511663f8d4f2d4fba41bfbf3c3fa0926376af609b976308" Nov 04 01:38:35.757812 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:38:35.757781 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4790695f45e48e3f2511663f8d4f2d4fba41bfbf3c3fa0926376af609b976308\": container with ID starting with 4790695f45e48e3f2511663f8d4f2d4fba41bfbf3c3fa0926376af609b976308 not found: ID does not exist" containerID="4790695f45e48e3f2511663f8d4f2d4fba41bfbf3c3fa0926376af609b976308" Nov 04 01:38:35.757920 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.757820 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4790695f45e48e3f2511663f8d4f2d4fba41bfbf3c3fa0926376af609b976308"} err="failed to get container status \"4790695f45e48e3f2511663f8d4f2d4fba41bfbf3c3fa0926376af609b976308\": rpc error: code = NotFound desc = could not find container \"4790695f45e48e3f2511663f8d4f2d4fba41bfbf3c3fa0926376af609b976308\": container with ID starting with 4790695f45e48e3f2511663f8d4f2d4fba41bfbf3c3fa0926376af609b976308 not found: ID does not exist" Nov 04 01:38:35.898351 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.898312 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp"] Nov 04 01:38:35.945288 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:35.945253 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d057d7/payload-processing-external-model-e2e-extmod-b-d057d7-8584mrbjp"] Nov 04 01:38:36.130508 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:36.130405 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0686885-ff19-476d-9bff-c5a02d5855ae" path="/var/lib/kubelet/pods/f0686885-ff19-476d-9bff-c5a02d5855ae/volumes" Nov 04 01:38:44.123842 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:44.123798 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:38:49.641835 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:38:49.641799 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:38:52.669914 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:52.669843 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:38:54.123931 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:54.123889 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:38:55.544193 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:55.544150 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s"] Nov 04 01:38:55.544869 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:55.544822 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" podUID="45cc4145-4d28-44b1-a76d-a24535f207e7" containerName="istio-proxy" containerID="cri-o://93fa42310d0c0bc7faec3e47bd3436d33433ad586dc150e272718696fa2eb3a7" gracePeriod=30 Nov 04 01:38:56.796581 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.796540 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq"] Nov 04 01:38:56.797209 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.797190 5995 cpu_manager.go:414] "containerMap: removing container" podUID="31857d37-58de-4a1b-a3bf-856b7f20c244" containerName="revoke-keys" Nov 04 01:38:56.797272 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.797211 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31857d37-58de-4a1b-a3bf-856b7f20c244" containerName="revoke-keys" Nov 04 01:38:56.797272 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.797253 5995 cpu_manager.go:414] "containerMap: removing container" podUID="1798ca5e-faea-498a-941a-c810c4c69e33" containerName="istio-proxy" Nov 04 01:38:56.797272 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.797259 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1798ca5e-faea-498a-941a-c810c4c69e33" containerName="istio-proxy" Nov 04 01:38:56.797272 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.797270 5995 cpu_manager.go:414] "containerMap: removing container" podUID="f0686885-ff19-476d-9bff-c5a02d5855ae" containerName="payload-processing" Nov 04 01:38:56.797397 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.797274 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0686885-ff19-476d-9bff-c5a02d5855ae" containerName="payload-processing" Nov 04 01:38:56.797397 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.797355 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="1798ca5e-faea-498a-941a-c810c4c69e33" containerName="istio-proxy" Nov 04 01:38:56.797397 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.797364 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0686885-ff19-476d-9bff-c5a02d5855ae" containerName="payload-processing" Nov 04 01:38:56.797397 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.797372 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="31857d37-58de-4a1b-a3bf-856b7f20c244" containerName="revoke-keys" Nov 04 01:38:56.802555 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.802537 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq" Nov 04 01:38:56.898527 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.898488 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:38:56.898705 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.898644 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w799x\"" Nov 04 01:38:56.898705 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.898689 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:38:56.898814 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.898768 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:38:56.917392 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.917362 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq"] Nov 04 01:38:56.993653 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.993616 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f88b\" (UniqueName: \"kubernetes.io/projected/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-kube-api-access-6f88b\") pod \"maas-api-revoke-sub-e80ca214-qchfq\" (UID: \"1dbdae25-2ea8-41b9-b38b-83ccda7f7a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq" Nov 04 01:38:56.993866 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.993771 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e80ca214-qchfq\" (UID: \"1dbdae25-2ea8-41b9-b38b-83ccda7f7a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq" Nov 04 01:38:56.993926 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:56.993893 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e80ca214-qchfq\" (UID: \"1dbdae25-2ea8-41b9-b38b-83ccda7f7a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq" Nov 04 01:38:57.094794 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:57.094697 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f88b\" (UniqueName: \"kubernetes.io/projected/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-kube-api-access-6f88b\") pod \"maas-api-revoke-sub-e80ca214-qchfq\" (UID: \"1dbdae25-2ea8-41b9-b38b-83ccda7f7a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq" Nov 04 01:38:57.094971 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:57.094849 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e80ca214-qchfq\" (UID: \"1dbdae25-2ea8-41b9-b38b-83ccda7f7a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq" Nov 04 01:38:57.094971 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:57.094910 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e80ca214-qchfq\" (UID: \"1dbdae25-2ea8-41b9-b38b-83ccda7f7a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq" Nov 04 01:38:57.095557 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:57.095535 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e80ca214-qchfq\" (UID: \"1dbdae25-2ea8-41b9-b38b-83ccda7f7a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq" Nov 04 01:38:57.097341 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:57.097315 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e80ca214-qchfq\" (UID: \"1dbdae25-2ea8-41b9-b38b-83ccda7f7a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq" Nov 04 01:38:57.156752 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:57.156723 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f88b\" (UniqueName: \"kubernetes.io/projected/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-kube-api-access-6f88b\") pod \"maas-api-revoke-sub-e80ca214-qchfq\" (UID: \"1dbdae25-2ea8-41b9-b38b-83ccda7f7a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq" Nov 04 01:38:57.417101 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:57.417019 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq" Nov 04 01:38:57.797949 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:57.797916 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq"] Nov 04 01:38:57.803381 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:38:57.803351 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbdae25_2ea8_41b9_b38b_83ccda7f7a65.slice/crio-0433c566fac34fb75b220351d1aaca2a79f48555dad3e70b2276f3b85f452d7c WatchSource:0}: Error finding container 0433c566fac34fb75b220351d1aaca2a79f48555dad3e70b2276f3b85f452d7c: Status 404 returned error can't find the container with id 0433c566fac34fb75b220351d1aaca2a79f48555dad3e70b2276f3b85f452d7c Nov 04 01:38:57.918313 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:57.918266 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq" event={"ID":"1dbdae25-2ea8-41b9-b38b-83ccda7f7a65","Type":"ContainerStarted","Data":"0433c566fac34fb75b220351d1aaca2a79f48555dad3e70b2276f3b85f452d7c"} Nov 04 01:38:58.926218 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:58.926136 5995 generic.go:356] "Generic (PLEG): container finished" podID="1dbdae25-2ea8-41b9-b38b-83ccda7f7a65" containerID="d85030d19f080e5a9afcd2877b55011eebd5643873acc49ac30372bf8bff028e" exitCode=0 Nov 04 01:38:58.926599 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:58.926224 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq" event={"ID":"1dbdae25-2ea8-41b9-b38b-83ccda7f7a65","Type":"ContainerDied","Data":"d85030d19f080e5a9afcd2877b55011eebd5643873acc49ac30372bf8bff028e"} Nov 04 01:38:59.472230 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:59.472196 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm"] Nov 04 01:38:59.580749 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:38:59.580717 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv"] Nov 04 01:39:00.169480 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.169454 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq" Nov 04 01:39:00.239136 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.239054 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-maas-api-xks-ca\") pod \"1dbdae25-2ea8-41b9-b38b-83ccda7f7a65\" (UID: \"1dbdae25-2ea8-41b9-b38b-83ccda7f7a65\") " Nov 04 01:39:00.239136 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.239127 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-maas-api-openshift-ca\") pod \"1dbdae25-2ea8-41b9-b38b-83ccda7f7a65\" (UID: \"1dbdae25-2ea8-41b9-b38b-83ccda7f7a65\") " Nov 04 01:39:00.239372 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.239209 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-kube-api-access-6f88b\" (UniqueName: \"kubernetes.io/projected/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-kube-api-access-6f88b\") pod \"1dbdae25-2ea8-41b9-b38b-83ccda7f7a65\" (UID: \"1dbdae25-2ea8-41b9-b38b-83ccda7f7a65\") " Nov 04 01:39:00.239603 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.239576 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-maas-api-openshift-ca" pod "1dbdae25-2ea8-41b9-b38b-83ccda7f7a65" (UID: "1dbdae25-2ea8-41b9-b38b-83ccda7f7a65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:00.239763 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.239744 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-maas-api-openshift-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:00.241408 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.241370 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-maas-api-xks-ca" pod "1dbdae25-2ea8-41b9-b38b-83ccda7f7a65" (UID: "1dbdae25-2ea8-41b9-b38b-83ccda7f7a65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:39:00.241408 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.241394 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-kube-api-access-6f88b" pod "1dbdae25-2ea8-41b9-b38b-83ccda7f7a65" (UID: "1dbdae25-2ea8-41b9-b38b-83ccda7f7a65"). InnerVolumeSpecName "kube-api-access-6f88b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:00.341164 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.341129 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f88b\" (UniqueName: \"kubernetes.io/projected/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-kube-api-access-6f88b\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:00.341164 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.341157 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65-maas-api-xks-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:00.792333 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.792305 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:39:00.846717 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.846638 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-workload-socket\") pod \"45cc4145-4d28-44b1-a76d-a24535f207e7\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " Nov 04 01:39:00.846717 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.846689 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-credential-socket\") pod \"45cc4145-4d28-44b1-a76d-a24535f207e7\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " Nov 04 01:39:00.846937 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.846725 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-podinfo\") pod \"45cc4145-4d28-44b1-a76d-a24535f207e7\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " Nov 04 01:39:00.846937 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.846758 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45cc4145-4d28-44b1-a76d-a24535f207e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45cc4145-4d28-44b1-a76d-a24535f207e7-istiod-ca-cert\") pod \"45cc4145-4d28-44b1-a76d-a24535f207e7\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " Nov 04 01:39:00.846937 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.846801 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45cc4145-4d28-44b1-a76d-a24535f207e7-kube-api-access-psqnt\" (UniqueName: \"kubernetes.io/projected/45cc4145-4d28-44b1-a76d-a24535f207e7-kube-api-access-psqnt\") pod \"45cc4145-4d28-44b1-a76d-a24535f207e7\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " Nov 04 01:39:00.846937 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.846861 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-data\") pod \"45cc4145-4d28-44b1-a76d-a24535f207e7\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " Nov 04 01:39:00.846937 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.846918 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-workload-socket" pod "45cc4145-4d28-44b1-a76d-a24535f207e7" (UID: "45cc4145-4d28-44b1-a76d-a24535f207e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:39:00.847195 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.846946 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-workload-certs\") pod \"45cc4145-4d28-44b1-a76d-a24535f207e7\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " Nov 04 01:39:00.847195 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.846999 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-envoy\") pod \"45cc4145-4d28-44b1-a76d-a24535f207e7\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " Nov 04 01:39:00.847195 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.847023 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-token\" (UniqueName: \"kubernetes.io/projected/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-token\") pod \"45cc4145-4d28-44b1-a76d-a24535f207e7\" (UID: \"45cc4145-4d28-44b1-a76d-a24535f207e7\") " Nov 04 01:39:00.847195 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.847041 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-credential-socket" pod "45cc4145-4d28-44b1-a76d-a24535f207e7" (UID: "45cc4145-4d28-44b1-a76d-a24535f207e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:39:00.847195 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.847151 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-data" pod "45cc4145-4d28-44b1-a76d-a24535f207e7" (UID: "45cc4145-4d28-44b1-a76d-a24535f207e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:39:00.847443 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.847195 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-workload-certs" pod "45cc4145-4d28-44b1-a76d-a24535f207e7" (UID: "45cc4145-4d28-44b1-a76d-a24535f207e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:39:00.847443 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.847230 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45cc4145-4d28-44b1-a76d-a24535f207e7-istiod-ca-cert" pod "45cc4145-4d28-44b1-a76d-a24535f207e7" (UID: "45cc4145-4d28-44b1-a76d-a24535f207e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:00.847579 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.847557 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:00.847639 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.847585 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:00.847639 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.847613 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45cc4145-4d28-44b1-a76d-a24535f207e7-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:00.847639 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.847626 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:00.847791 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.847639 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:00.849054 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.849030 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-envoy" pod "45cc4145-4d28-44b1-a76d-a24535f207e7" (UID: "45cc4145-4d28-44b1-a76d-a24535f207e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:39:00.849054 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.849049 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-podinfo" pod "45cc4145-4d28-44b1-a76d-a24535f207e7" (UID: "45cc4145-4d28-44b1-a76d-a24535f207e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:39:00.849207 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.849061 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45cc4145-4d28-44b1-a76d-a24535f207e7-kube-api-access-psqnt" pod "45cc4145-4d28-44b1-a76d-a24535f207e7" (UID: "45cc4145-4d28-44b1-a76d-a24535f207e7"). InnerVolumeSpecName "kube-api-access-psqnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:00.849207 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.849079 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-token" pod "45cc4145-4d28-44b1-a76d-a24535f207e7" (UID: "45cc4145-4d28-44b1-a76d-a24535f207e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:00.969439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.968469 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:00.969439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.968511 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:00.969439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.968526 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45cc4145-4d28-44b1-a76d-a24535f207e7-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:00.969439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.968540 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psqnt\" (UniqueName: \"kubernetes.io/projected/45cc4145-4d28-44b1-a76d-a24535f207e7-kube-api-access-psqnt\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:00.983357 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.983320 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq" event={"ID":"1dbdae25-2ea8-41b9-b38b-83ccda7f7a65","Type":"ContainerDied","Data":"0433c566fac34fb75b220351d1aaca2a79f48555dad3e70b2276f3b85f452d7c"} Nov 04 01:39:00.983570 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.983408 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0433c566fac34fb75b220351d1aaca2a79f48555dad3e70b2276f3b85f452d7c" Nov 04 01:39:00.983570 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.983492 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq" Nov 04 01:39:00.987807 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.987760 5995 generic.go:356] "Generic (PLEG): container finished" podID="45cc4145-4d28-44b1-a76d-a24535f207e7" containerID="93fa42310d0c0bc7faec3e47bd3436d33433ad586dc150e272718696fa2eb3a7" exitCode=0 Nov 04 01:39:00.988203 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.988138 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" event={"ID":"45cc4145-4d28-44b1-a76d-a24535f207e7","Type":"ContainerDied","Data":"93fa42310d0c0bc7faec3e47bd3436d33433ad586dc150e272718696fa2eb3a7"} Nov 04 01:39:00.988409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.988346 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" event={"ID":"45cc4145-4d28-44b1-a76d-a24535f207e7","Type":"ContainerDied","Data":"98bf916bf1ff4e0f2b9a0ba1aa9091616fcbb0c5884a130affecdc6fbe48c95a"} Nov 04 01:39:00.988409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.988389 5995 scope.go:122] "RemoveContainer" containerID="93fa42310d0c0bc7faec3e47bd3436d33433ad586dc150e272718696fa2eb3a7" Nov 04 01:39:00.990440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:00.989099 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s" Nov 04 01:39:01.005745 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:01.005724 5995 scope.go:122] "RemoveContainer" containerID="93fa42310d0c0bc7faec3e47bd3436d33433ad586dc150e272718696fa2eb3a7" Nov 04 01:39:01.006070 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:39:01.006052 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93fa42310d0c0bc7faec3e47bd3436d33433ad586dc150e272718696fa2eb3a7\": container with ID starting with 93fa42310d0c0bc7faec3e47bd3436d33433ad586dc150e272718696fa2eb3a7 not found: ID does not exist" containerID="93fa42310d0c0bc7faec3e47bd3436d33433ad586dc150e272718696fa2eb3a7" Nov 04 01:39:01.006109 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:01.006081 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93fa42310d0c0bc7faec3e47bd3436d33433ad586dc150e272718696fa2eb3a7"} err="failed to get container status \"93fa42310d0c0bc7faec3e47bd3436d33433ad586dc150e272718696fa2eb3a7\": rpc error: code = NotFound desc = could not find container \"93fa42310d0c0bc7faec3e47bd3436d33433ad586dc150e272718696fa2eb3a7\": container with ID starting with 93fa42310d0c0bc7faec3e47bd3436d33433ad586dc150e272718696fa2eb3a7 not found: ID does not exist" Nov 04 01:39:01.154729 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:01.154698 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s"] Nov 04 01:39:01.160075 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:01.160041 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-nl22s"] Nov 04 01:39:02.129690 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:02.129657 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45cc4145-4d28-44b1-a76d-a24535f207e7" path="/var/lib/kubelet/pods/45cc4145-4d28-44b1-a76d-a24535f207e7/volumes" Nov 04 01:39:04.123622 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:04.123591 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:39:04.473336 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:04.473274 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" podUID="384eb14f-2381-4803-87fb-9c2a6ed88529" containerName="payload-processing" containerID="cri-o://c0098b4bb062760369f3a2c57021aed792dcc766fd28e986fe8c8883b244d64f" gracePeriod=25 Nov 04 01:39:04.586475 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:04.586433 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" podUID="07dd8e8d-fed9-42fa-a391-dc66bb56a1a2" containerName="payload-pre-processing" containerID="cri-o://4f4739090cde7fcc25b024fb9bc167d5a3d1685b7c70880e26cf82b24644655a" gracePeriod=25 Nov 04 01:39:10.211905 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:10.211869 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h"] Nov 04 01:39:11.499910 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:11.499876 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" podUID="07dd8e8d-fed9-42fa-a391-dc66bb56a1a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:39:11.500261 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:11.499877 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" podUID="384eb14f-2381-4803-87fb-9c2a6ed88529" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:39:14.124594 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:14.124556 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:39:15.212808 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:15.212756 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" podUID="b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629" containerName="payload-processing" containerID="cri-o://a9d362002b5f99571eebaf846480a0ccefbddd1db269ae2e97c0d60d7a6b93be" gracePeriod=25 Nov 04 01:39:15.480666 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:15.480598 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" Nov 04 01:39:15.588027 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:15.587992 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-routing-overlay-volume\") pod \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\" (UID: \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\") " Nov 04 01:39:15.588027 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:15.588034 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-kube-api-access-dqgf5\" (UniqueName: \"kubernetes.io/projected/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-kube-api-access-dqgf5\") pod \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\" (UID: \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\") " Nov 04 01:39:15.588248 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:15.588058 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-provider-credentials\") pod \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\" (UID: \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\") " Nov 04 01:39:15.588248 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:15.588201 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-plugins-config-volume\") pod \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\" (UID: \"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629\") " Nov 04 01:39:15.588383 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:15.588356 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-routing-overlay-volume" pod "b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629" (UID: "b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:15.588642 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:15.588618 5995 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-routing-overlay-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:15.588756 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:15.588643 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-plugins-config-volume" pod "b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629" (UID: "b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:15.590445 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:15.590408 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-provider-credentials" pod "b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629" (UID: "b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:15.590568 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:15.590546 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-kube-api-access-dqgf5" pod "b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629" (UID: "b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629"). InnerVolumeSpecName "kube-api-access-dqgf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:15.690348 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:15.690314 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:15.690348 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:15.690344 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqgf5\" (UniqueName: \"kubernetes.io/projected/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-kube-api-access-dqgf5\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:15.690348 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:15.690353 5995 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629-provider-credentials\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:16.121301 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:16.121261 5995 generic.go:356] "Generic (PLEG): container finished" podID="b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629" containerID="a9d362002b5f99571eebaf846480a0ccefbddd1db269ae2e97c0d60d7a6b93be" exitCode=0 Nov 04 01:39:16.121513 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:16.121320 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" Nov 04 01:39:16.121513 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:16.121336 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" event={"ID":"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629","Type":"ContainerDied","Data":"a9d362002b5f99571eebaf846480a0ccefbddd1db269ae2e97c0d60d7a6b93be"} Nov 04 01:39:16.132103 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:16.132075 5995 scope.go:122] "RemoveContainer" containerID="a9d362002b5f99571eebaf846480a0ccefbddd1db269ae2e97c0d60d7a6b93be" Nov 04 01:39:16.138884 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:16.138850 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h" event={"ID":"b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629","Type":"ContainerDied","Data":"22e74e9475ba168e084c85913a2faa9fade55c2290370e88726b48c84f1c0a75"} Nov 04 01:39:16.146009 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:16.145982 5995 scope.go:122] "RemoveContainer" containerID="a9d362002b5f99571eebaf846480a0ccefbddd1db269ae2e97c0d60d7a6b93be" Nov 04 01:39:16.146402 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:39:16.146372 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d362002b5f99571eebaf846480a0ccefbddd1db269ae2e97c0d60d7a6b93be\": container with ID starting with a9d362002b5f99571eebaf846480a0ccefbddd1db269ae2e97c0d60d7a6b93be not found: ID does not exist" containerID="a9d362002b5f99571eebaf846480a0ccefbddd1db269ae2e97c0d60d7a6b93be" Nov 04 01:39:16.146575 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:16.146427 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d362002b5f99571eebaf846480a0ccefbddd1db269ae2e97c0d60d7a6b93be"} err="failed to get container status \"a9d362002b5f99571eebaf846480a0ccefbddd1db269ae2e97c0d60d7a6b93be\": rpc error: code = NotFound desc = could not find container \"a9d362002b5f99571eebaf846480a0ccefbddd1db269ae2e97c0d60d7a6b93be\": container with ID starting with a9d362002b5f99571eebaf846480a0ccefbddd1db269ae2e97c0d60d7a6b93be not found: ID does not exist" Nov 04 01:39:16.276891 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:16.276856 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h"] Nov 04 01:39:16.323216 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:16.323180 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1225f3/payload-processing-external-model-e2e-extmod-b-1225f3-6d8fp6b2h"] Nov 04 01:39:18.129383 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:18.129349 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629" path="/var/lib/kubelet/pods/b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629/volumes" Nov 04 01:39:19.641885 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:39:19.641854 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:39:20.283978 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:20.283939 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:39:21.499493 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:21.499453 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" podUID="07dd8e8d-fed9-42fa-a391-dc66bb56a1a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:39:21.499493 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:21.499488 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" podUID="384eb14f-2381-4803-87fb-9c2a6ed88529" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:39:24.124286 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:24.124247 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:39:24.753721 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:24.753647 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" Nov 04 01:39:24.862675 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:24.862642 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" Nov 04 01:39:24.910731 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:24.910696 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384eb14f-2381-4803-87fb-9c2a6ed88529-kube-api-access-str2s\" (UniqueName: \"kubernetes.io/projected/384eb14f-2381-4803-87fb-9c2a6ed88529-kube-api-access-str2s\") pod \"384eb14f-2381-4803-87fb-9c2a6ed88529\" (UID: \"384eb14f-2381-4803-87fb-9c2a6ed88529\") " Nov 04 01:39:24.910924 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:24.910902 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/384eb14f-2381-4803-87fb-9c2a6ed88529-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/384eb14f-2381-4803-87fb-9c2a6ed88529-plugins-config-volume\") pod \"384eb14f-2381-4803-87fb-9c2a6ed88529\" (UID: \"384eb14f-2381-4803-87fb-9c2a6ed88529\") " Nov 04 01:39:24.911329 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:24.911297 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/384eb14f-2381-4803-87fb-9c2a6ed88529-plugins-config-volume" pod "384eb14f-2381-4803-87fb-9c2a6ed88529" (UID: "384eb14f-2381-4803-87fb-9c2a6ed88529"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:24.912907 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:24.912883 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384eb14f-2381-4803-87fb-9c2a6ed88529-kube-api-access-str2s" pod "384eb14f-2381-4803-87fb-9c2a6ed88529" (UID: "384eb14f-2381-4803-87fb-9c2a6ed88529"). InnerVolumeSpecName "kube-api-access-str2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:25.012730 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.012625 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07dd8e8d-fed9-42fa-a391-dc66bb56a1a2-kube-api-access-tjhg8\" (UniqueName: \"kubernetes.io/projected/07dd8e8d-fed9-42fa-a391-dc66bb56a1a2-kube-api-access-tjhg8\") pod \"07dd8e8d-fed9-42fa-a391-dc66bb56a1a2\" (UID: \"07dd8e8d-fed9-42fa-a391-dc66bb56a1a2\") " Nov 04 01:39:25.012911 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.012797 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07dd8e8d-fed9-42fa-a391-dc66bb56a1a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07dd8e8d-fed9-42fa-a391-dc66bb56a1a2-plugins-config-volume\") pod \"07dd8e8d-fed9-42fa-a391-dc66bb56a1a2\" (UID: \"07dd8e8d-fed9-42fa-a391-dc66bb56a1a2\") " Nov 04 01:39:25.013347 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.013318 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-str2s\" (UniqueName: \"kubernetes.io/projected/384eb14f-2381-4803-87fb-9c2a6ed88529-kube-api-access-str2s\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:25.013498 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.013350 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/384eb14f-2381-4803-87fb-9c2a6ed88529-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:25.039440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.032316 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07dd8e8d-fed9-42fa-a391-dc66bb56a1a2-plugins-config-volume" pod "07dd8e8d-fed9-42fa-a391-dc66bb56a1a2" (UID: "07dd8e8d-fed9-42fa-a391-dc66bb56a1a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:25.083637 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.083593 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07dd8e8d-fed9-42fa-a391-dc66bb56a1a2-kube-api-access-tjhg8" pod "07dd8e8d-fed9-42fa-a391-dc66bb56a1a2" (UID: "07dd8e8d-fed9-42fa-a391-dc66bb56a1a2"). InnerVolumeSpecName "kube-api-access-tjhg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:25.123438 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.123389 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjhg8\" (UniqueName: \"kubernetes.io/projected/07dd8e8d-fed9-42fa-a391-dc66bb56a1a2-kube-api-access-tjhg8\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:25.126149 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.126109 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07dd8e8d-fed9-42fa-a391-dc66bb56a1a2-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:25.261909 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.261872 5995 generic.go:356] "Generic (PLEG): container finished" podID="384eb14f-2381-4803-87fb-9c2a6ed88529" containerID="c0098b4bb062760369f3a2c57021aed792dcc766fd28e986fe8c8883b244d64f" exitCode=0 Nov 04 01:39:25.262105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.261932 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" Nov 04 01:39:25.262105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.261953 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" event={"ID":"384eb14f-2381-4803-87fb-9c2a6ed88529","Type":"ContainerDied","Data":"c0098b4bb062760369f3a2c57021aed792dcc766fd28e986fe8c8883b244d64f"} Nov 04 01:39:25.262105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.262014 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm" event={"ID":"384eb14f-2381-4803-87fb-9c2a6ed88529","Type":"ContainerDied","Data":"ed141acc873a1969e69b4a5fb955c59d4b815e15956034870c6f65f9c3b4b63c"} Nov 04 01:39:25.262273 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.262115 5995 scope.go:122] "RemoveContainer" containerID="c0098b4bb062760369f3a2c57021aed792dcc766fd28e986fe8c8883b244d64f" Nov 04 01:39:25.264080 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.264050 5995 generic.go:356] "Generic (PLEG): container finished" podID="07dd8e8d-fed9-42fa-a391-dc66bb56a1a2" containerID="4f4739090cde7fcc25b024fb9bc167d5a3d1685b7c70880e26cf82b24644655a" exitCode=0 Nov 04 01:39:25.264199 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.264184 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" event={"ID":"07dd8e8d-fed9-42fa-a391-dc66bb56a1a2","Type":"ContainerDied","Data":"4f4739090cde7fcc25b024fb9bc167d5a3d1685b7c70880e26cf82b24644655a"} Nov 04 01:39:25.264259 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.264224 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" event={"ID":"07dd8e8d-fed9-42fa-a391-dc66bb56a1a2","Type":"ContainerDied","Data":"2452e8d91bde2aec573891e8999458a5a96542044b02c1ce7335c820bd0aac68"} Nov 04 01:39:25.264326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.264313 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv" Nov 04 01:39:25.275857 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.275839 5995 scope.go:122] "RemoveContainer" containerID="c0098b4bb062760369f3a2c57021aed792dcc766fd28e986fe8c8883b244d64f" Nov 04 01:39:25.276120 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:39:25.276103 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0098b4bb062760369f3a2c57021aed792dcc766fd28e986fe8c8883b244d64f\": container with ID starting with c0098b4bb062760369f3a2c57021aed792dcc766fd28e986fe8c8883b244d64f not found: ID does not exist" containerID="c0098b4bb062760369f3a2c57021aed792dcc766fd28e986fe8c8883b244d64f" Nov 04 01:39:25.276187 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.276128 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0098b4bb062760369f3a2c57021aed792dcc766fd28e986fe8c8883b244d64f"} err="failed to get container status \"c0098b4bb062760369f3a2c57021aed792dcc766fd28e986fe8c8883b244d64f\": rpc error: code = NotFound desc = could not find container \"c0098b4bb062760369f3a2c57021aed792dcc766fd28e986fe8c8883b244d64f\": container with ID starting with c0098b4bb062760369f3a2c57021aed792dcc766fd28e986fe8c8883b244d64f not found: ID does not exist" Nov 04 01:39:25.276187 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.276145 5995 scope.go:122] "RemoveContainer" containerID="4f4739090cde7fcc25b024fb9bc167d5a3d1685b7c70880e26cf82b24644655a" Nov 04 01:39:25.286066 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.286040 5995 scope.go:122] "RemoveContainer" containerID="4f4739090cde7fcc25b024fb9bc167d5a3d1685b7c70880e26cf82b24644655a" Nov 04 01:39:25.286372 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:39:25.286346 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f4739090cde7fcc25b024fb9bc167d5a3d1685b7c70880e26cf82b24644655a\": container with ID starting with 4f4739090cde7fcc25b024fb9bc167d5a3d1685b7c70880e26cf82b24644655a not found: ID does not exist" containerID="4f4739090cde7fcc25b024fb9bc167d5a3d1685b7c70880e26cf82b24644655a" Nov 04 01:39:25.286470 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.286381 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f4739090cde7fcc25b024fb9bc167d5a3d1685b7c70880e26cf82b24644655a"} err="failed to get container status \"4f4739090cde7fcc25b024fb9bc167d5a3d1685b7c70880e26cf82b24644655a\": rpc error: code = NotFound desc = could not find container \"4f4739090cde7fcc25b024fb9bc167d5a3d1685b7c70880e26cf82b24644655a\": container with ID starting with 4f4739090cde7fcc25b024fb9bc167d5a3d1685b7c70880e26cf82b24644655a not found: ID does not exist" Nov 04 01:39:25.420971 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.419045 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm"] Nov 04 01:39:25.464160 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.464120 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6db2a7-5884755cfb-td2rm"] Nov 04 01:39:25.533309 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.533215 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv"] Nov 04 01:39:25.539512 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:25.539485 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6db2a7-549f767475-pltcv"] Nov 04 01:39:26.129680 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:26.129650 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07dd8e8d-fed9-42fa-a391-dc66bb56a1a2" path="/var/lib/kubelet/pods/07dd8e8d-fed9-42fa-a391-dc66bb56a1a2/volumes" Nov 04 01:39:26.130060 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:26.130044 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="384eb14f-2381-4803-87fb-9c2a6ed88529" path="/var/lib/kubelet/pods/384eb14f-2381-4803-87fb-9c2a6ed88529/volumes" Nov 04 01:39:29.058492 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.058319 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm"] Nov 04 01:39:29.071548 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.071508 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:39:29.096177 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.095956 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7789777d6-l8vds"] Nov 04 01:39:29.096774 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.096749 5995 cpu_manager.go:414] "containerMap: removing container" podUID="b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629" containerName="payload-processing" Nov 04 01:39:29.096774 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.096775 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629" containerName="payload-processing" Nov 04 01:39:29.096959 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.096803 5995 cpu_manager.go:414] "containerMap: removing container" podUID="45cc4145-4d28-44b1-a76d-a24535f207e7" containerName="istio-proxy" Nov 04 01:39:29.096959 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.096811 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45cc4145-4d28-44b1-a76d-a24535f207e7" containerName="istio-proxy" Nov 04 01:39:29.096959 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.096835 5995 cpu_manager.go:414] "containerMap: removing container" podUID="07dd8e8d-fed9-42fa-a391-dc66bb56a1a2" containerName="payload-pre-processing" Nov 04 01:39:29.096959 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.096842 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07dd8e8d-fed9-42fa-a391-dc66bb56a1a2" containerName="payload-pre-processing" Nov 04 01:39:29.096959 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.096858 5995 cpu_manager.go:414] "containerMap: removing container" podUID="384eb14f-2381-4803-87fb-9c2a6ed88529" containerName="payload-processing" Nov 04 01:39:29.096959 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.096865 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384eb14f-2381-4803-87fb-9c2a6ed88529" containerName="payload-processing" Nov 04 01:39:29.096959 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.096897 5995 cpu_manager.go:414] "containerMap: removing container" podUID="1dbdae25-2ea8-41b9-b38b-83ccda7f7a65" containerName="revoke-keys" Nov 04 01:39:29.096959 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.096903 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dbdae25-2ea8-41b9-b38b-83ccda7f7a65" containerName="revoke-keys" Nov 04 01:39:29.097324 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.097045 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5fadc3a-fea6-4a2a-b0e6-f197ce2ef629" containerName="payload-processing" Nov 04 01:39:29.097324 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.097062 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="45cc4145-4d28-44b1-a76d-a24535f207e7" containerName="istio-proxy" Nov 04 01:39:29.097324 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.097074 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="07dd8e8d-fed9-42fa-a391-dc66bb56a1a2" containerName="payload-pre-processing" Nov 04 01:39:29.097324 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.097085 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="384eb14f-2381-4803-87fb-9c2a6ed88529" containerName="payload-processing" Nov 04 01:39:29.097324 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.097096 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dbdae25-2ea8-41b9-b38b-83ccda7f7a65" containerName="revoke-keys" Nov 04 01:39:29.103809 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.103782 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" Nov 04 01:39:29.123289 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.123259 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials\") pod \"payload-processing-external-model-7789777d6-l8vds\" (UID: \"95384e76-af24-412c-b18e-2177d638c7d4\") " pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" Nov 04 01:39:29.123453 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.123423 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95384e76-af24-412c-b18e-2177d638c7d4-routing-overlay-volume\") pod \"payload-processing-external-model-7789777d6-l8vds\" (UID: \"95384e76-af24-412c-b18e-2177d638c7d4\") " pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" Nov 04 01:39:29.123522 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.123472 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95384e76-af24-412c-b18e-2177d638c7d4-plugins-config-volume\") pod \"payload-processing-external-model-7789777d6-l8vds\" (UID: \"95384e76-af24-412c-b18e-2177d638c7d4\") " pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" Nov 04 01:39:29.123522 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.123500 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2fzn\" (UniqueName: \"kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-kube-api-access-p2fzn\") pod \"payload-processing-external-model-7789777d6-l8vds\" (UID: \"95384e76-af24-412c-b18e-2177d638c7d4\") " pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" Nov 04 01:39:29.153674 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.153644 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-5e61507d7978\"" Nov 04 01:39:29.155596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.155566 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7789777d6-l8vds"] Nov 04 01:39:29.224672 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.224636 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95384e76-af24-412c-b18e-2177d638c7d4-routing-overlay-volume\") pod \"payload-processing-external-model-7789777d6-l8vds\" (UID: \"95384e76-af24-412c-b18e-2177d638c7d4\") " pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" Nov 04 01:39:29.224867 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.224679 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95384e76-af24-412c-b18e-2177d638c7d4-plugins-config-volume\") pod \"payload-processing-external-model-7789777d6-l8vds\" (UID: \"95384e76-af24-412c-b18e-2177d638c7d4\") " pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" Nov 04 01:39:29.224867 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.224704 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2fzn\" (UniqueName: \"kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-kube-api-access-p2fzn\") pod \"payload-processing-external-model-7789777d6-l8vds\" (UID: \"95384e76-af24-412c-b18e-2177d638c7d4\") " pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" Nov 04 01:39:29.224867 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.224784 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials\") pod \"payload-processing-external-model-7789777d6-l8vds\" (UID: \"95384e76-af24-412c-b18e-2177d638c7d4\") " pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" Nov 04 01:39:29.225342 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.225315 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95384e76-af24-412c-b18e-2177d638c7d4-routing-overlay-volume\") pod \"payload-processing-external-model-7789777d6-l8vds\" (UID: \"95384e76-af24-412c-b18e-2177d638c7d4\") " pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" Nov 04 01:39:29.225541 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.225520 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95384e76-af24-412c-b18e-2177d638c7d4-plugins-config-volume\") pod \"payload-processing-external-model-7789777d6-l8vds\" (UID: \"95384e76-af24-412c-b18e-2177d638c7d4\") " pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" Nov 04 01:39:29.227201 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.227181 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials\") pod \"payload-processing-external-model-7789777d6-l8vds\" (UID: \"95384e76-af24-412c-b18e-2177d638c7d4\") " pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" Nov 04 01:39:29.284968 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.284933 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2fzn\" (UniqueName: \"kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-kube-api-access-p2fzn\") pod \"payload-processing-external-model-7789777d6-l8vds\" (UID: \"95384e76-af24-412c-b18e-2177d638c7d4\") " pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" Nov 04 01:39:29.419994 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.419911 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" Nov 04 01:39:29.825027 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:29.824997 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7789777d6-l8vds"] Nov 04 01:39:29.828991 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:39:29.828963 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95384e76_af24_412c_b18e_2177d638c7d4.slice/crio-b1fa9ed5c33796ede2022f74870c96a221eb537322cc9dea907680d661b4a58f WatchSource:0}: Error finding container b1fa9ed5c33796ede2022f74870c96a221eb537322cc9dea907680d661b4a58f: Status 404 returned error can't find the container with id b1fa9ed5c33796ede2022f74870c96a221eb537322cc9dea907680d661b4a58f Nov 04 01:39:30.316673 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:30.316636 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" event={"ID":"95384e76-af24-412c-b18e-2177d638c7d4","Type":"ContainerStarted","Data":"1de25db9fb1c10689b90eca9b0c7b5d633e7aecb769131cd73d4eb4641dc37ee"} Nov 04 01:39:30.317071 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:30.316697 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" event={"ID":"95384e76-af24-412c-b18e-2177d638c7d4","Type":"ContainerStarted","Data":"b1fa9ed5c33796ede2022f74870c96a221eb537322cc9dea907680d661b4a58f"} Nov 04 01:39:30.317071 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:30.316754 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" Nov 04 01:39:30.432890 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:30.432841 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" podStartSLOduration=1.432826428 podStartE2EDuration="1.432826428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:39:30.429404655 +0000 UTC m=+1720.461138047" watchObservedRunningTime="2026-11-04 01:39:30.432826428 +0000 UTC m=+1720.464559817" Nov 04 01:39:32.110456 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:32.109468 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s"] Nov 04 01:39:32.110456 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:32.109724 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" podUID="d0cbdf70-f15f-42bf-b3f0-929212551be1" containerName="istio-proxy" containerID="cri-o://32ed93e35a841b8b2f673d534fcfc4f5514ce1bbf79435aa90f2256cf31a0167" gracePeriod=30 Nov 04 01:39:34.061590 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:34.061485 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" containerID="cri-o://59be1b727bb9d0b664897d05e1539b08648cc3d290ee4f0c5ef8feb53e9db84c" gracePeriod=25 Nov 04 01:39:34.432440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:34.431648 5995 generic.go:356] "Generic (PLEG): container finished" podID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerID="59be1b727bb9d0b664897d05e1539b08648cc3d290ee4f0c5ef8feb53e9db84c" exitCode=0 Nov 04 01:39:34.432440 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:34.431899 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" event={"ID":"c40ae805-9960-4a82-8ba6-b39b6bc6b458","Type":"ContainerDied","Data":"59be1b727bb9d0b664897d05e1539b08648cc3d290ee4f0c5ef8feb53e9db84c"} Nov 04 01:39:35.173645 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:35.173609 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" Nov 04 01:39:35.310435 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:35.306782 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40ae805-9960-4a82-8ba6-b39b6bc6b458-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c40ae805-9960-4a82-8ba6-b39b6bc6b458-routing-overlay-volume\") pod \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\" (UID: \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\") " Nov 04 01:39:35.310435 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:35.306933 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40ae805-9960-4a82-8ba6-b39b6bc6b458-kube-api-access-xkcf4\" (UniqueName: \"kubernetes.io/projected/c40ae805-9960-4a82-8ba6-b39b6bc6b458-kube-api-access-xkcf4\") pod \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\" (UID: \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\") " Nov 04 01:39:35.310435 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:35.307054 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40ae805-9960-4a82-8ba6-b39b6bc6b458-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c40ae805-9960-4a82-8ba6-b39b6bc6b458-provider-credentials\") pod \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\" (UID: \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\") " Nov 04 01:39:35.310435 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:35.307084 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40ae805-9960-4a82-8ba6-b39b6bc6b458-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c40ae805-9960-4a82-8ba6-b39b6bc6b458-plugins-config-volume\") pod \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\" (UID: \"c40ae805-9960-4a82-8ba6-b39b6bc6b458\") " Nov 04 01:39:35.310435 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:35.307842 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40ae805-9960-4a82-8ba6-b39b6bc6b458-plugins-config-volume" pod "c40ae805-9960-4a82-8ba6-b39b6bc6b458" (UID: "c40ae805-9960-4a82-8ba6-b39b6bc6b458"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:35.310435 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:35.308194 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40ae805-9960-4a82-8ba6-b39b6bc6b458-routing-overlay-volume" pod "c40ae805-9960-4a82-8ba6-b39b6bc6b458" (UID: "c40ae805-9960-4a82-8ba6-b39b6bc6b458"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:35.312106 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:35.312066 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40ae805-9960-4a82-8ba6-b39b6bc6b458-kube-api-access-xkcf4" pod "c40ae805-9960-4a82-8ba6-b39b6bc6b458" (UID: "c40ae805-9960-4a82-8ba6-b39b6bc6b458"). InnerVolumeSpecName "kube-api-access-xkcf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:35.368958 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:35.358946 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40ae805-9960-4a82-8ba6-b39b6bc6b458-provider-credentials" pod "c40ae805-9960-4a82-8ba6-b39b6bc6b458" (UID: "c40ae805-9960-4a82-8ba6-b39b6bc6b458"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:35.408865 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:35.408829 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkcf4\" (UniqueName: \"kubernetes.io/projected/c40ae805-9960-4a82-8ba6-b39b6bc6b458-kube-api-access-xkcf4\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:35.409095 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:35.409076 5995 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c40ae805-9960-4a82-8ba6-b39b6bc6b458-provider-credentials\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:35.409199 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:35.409186 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c40ae805-9960-4a82-8ba6-b39b6bc6b458-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:35.409296 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:35.409279 5995 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c40ae805-9960-4a82-8ba6-b39b6bc6b458-routing-overlay-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:35.520452 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:35.498928 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" event={"ID":"c40ae805-9960-4a82-8ba6-b39b6bc6b458","Type":"ContainerDied","Data":"646410d5c0f7e3475c1032b4f508cb914a316e89d9a914f90b907883ad1714a8"} Nov 04 01:39:35.520452 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:35.499001 5995 scope.go:122] "RemoveContainer" containerID="59be1b727bb9d0b664897d05e1539b08648cc3d290ee4f0c5ef8feb53e9db84c" Nov 04 01:39:35.520452 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:35.499157 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm" Nov 04 01:39:35.723280 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:35.723250 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm"] Nov 04 01:39:35.762400 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:35.762363 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dddfc6875-9n4rm"] Nov 04 01:39:36.133187 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:36.133096 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" path="/var/lib/kubelet/pods/c40ae805-9960-4a82-8ba6-b39b6bc6b458/volumes" Nov 04 01:39:37.312723 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.312695 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:39:37.353453 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.353395 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-podinfo\") pod \"d0cbdf70-f15f-42bf-b3f0-929212551be1\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " Nov 04 01:39:37.353631 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.353516 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-envoy\") pod \"d0cbdf70-f15f-42bf-b3f0-929212551be1\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " Nov 04 01:39:37.353631 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.353548 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-data\") pod \"d0cbdf70-f15f-42bf-b3f0-929212551be1\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " Nov 04 01:39:37.353631 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.353601 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-token\" (UniqueName: \"kubernetes.io/projected/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-token\") pod \"d0cbdf70-f15f-42bf-b3f0-929212551be1\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " Nov 04 01:39:37.353792 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.353647 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0cbdf70-f15f-42bf-b3f0-929212551be1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0cbdf70-f15f-42bf-b3f0-929212551be1-istiod-ca-cert\") pod \"d0cbdf70-f15f-42bf-b3f0-929212551be1\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " Nov 04 01:39:37.353792 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.353695 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0cbdf70-f15f-42bf-b3f0-929212551be1-kube-api-access-tqgj7\" (UniqueName: \"kubernetes.io/projected/d0cbdf70-f15f-42bf-b3f0-929212551be1-kube-api-access-tqgj7\") pod \"d0cbdf70-f15f-42bf-b3f0-929212551be1\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " Nov 04 01:39:37.353792 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.353727 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-workload-certs\") pod \"d0cbdf70-f15f-42bf-b3f0-929212551be1\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " Nov 04 01:39:37.353792 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.353778 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-credential-socket\") pod \"d0cbdf70-f15f-42bf-b3f0-929212551be1\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " Nov 04 01:39:37.353981 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.353828 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-workload-socket\") pod \"d0cbdf70-f15f-42bf-b3f0-929212551be1\" (UID: \"d0cbdf70-f15f-42bf-b3f0-929212551be1\") " Nov 04 01:39:37.353981 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.353860 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-data" pod "d0cbdf70-f15f-42bf-b3f0-929212551be1" (UID: "d0cbdf70-f15f-42bf-b3f0-929212551be1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:39:37.354092 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.354073 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-workload-certs" pod "d0cbdf70-f15f-42bf-b3f0-929212551be1" (UID: "d0cbdf70-f15f-42bf-b3f0-929212551be1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:39:37.354144 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.354112 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0cbdf70-f15f-42bf-b3f0-929212551be1-istiod-ca-cert" pod "d0cbdf70-f15f-42bf-b3f0-929212551be1" (UID: "d0cbdf70-f15f-42bf-b3f0-929212551be1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:37.354267 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.354243 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-credential-socket" pod "d0cbdf70-f15f-42bf-b3f0-929212551be1" (UID: "d0cbdf70-f15f-42bf-b3f0-929212551be1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:39:37.354364 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.354253 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-workload-socket" pod "d0cbdf70-f15f-42bf-b3f0-929212551be1" (UID: "d0cbdf70-f15f-42bf-b3f0-929212551be1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:39:37.354364 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.354330 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:37.354364 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.354348 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0cbdf70-f15f-42bf-b3f0-929212551be1-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:37.354364 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.354361 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:37.355698 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.355672 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-envoy" pod "d0cbdf70-f15f-42bf-b3f0-929212551be1" (UID: "d0cbdf70-f15f-42bf-b3f0-929212551be1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:39:37.355798 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.355739 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-podinfo" pod "d0cbdf70-f15f-42bf-b3f0-929212551be1" (UID: "d0cbdf70-f15f-42bf-b3f0-929212551be1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:39:37.356094 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.356078 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-token" pod "d0cbdf70-f15f-42bf-b3f0-929212551be1" (UID: "d0cbdf70-f15f-42bf-b3f0-929212551be1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:37.356158 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.356132 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0cbdf70-f15f-42bf-b3f0-929212551be1-kube-api-access-tqgj7" pod "d0cbdf70-f15f-42bf-b3f0-929212551be1" (UID: "d0cbdf70-f15f-42bf-b3f0-929212551be1"). InnerVolumeSpecName "kube-api-access-tqgj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:37.455523 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.455494 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:37.455523 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.455522 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:37.455729 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.455533 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqgj7\" (UniqueName: \"kubernetes.io/projected/d0cbdf70-f15f-42bf-b3f0-929212551be1-kube-api-access-tqgj7\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:37.455729 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.455543 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:37.455729 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.455552 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0cbdf70-f15f-42bf-b3f0-929212551be1-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:37.455729 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.455561 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0cbdf70-f15f-42bf-b3f0-929212551be1-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:37.514709 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.514670 5995 generic.go:356] "Generic (PLEG): container finished" podID="d0cbdf70-f15f-42bf-b3f0-929212551be1" containerID="32ed93e35a841b8b2f673d534fcfc4f5514ce1bbf79435aa90f2256cf31a0167" exitCode=0 Nov 04 01:39:37.514873 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.514738 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" Nov 04 01:39:37.514873 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.514753 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" event={"ID":"d0cbdf70-f15f-42bf-b3f0-929212551be1","Type":"ContainerDied","Data":"32ed93e35a841b8b2f673d534fcfc4f5514ce1bbf79435aa90f2256cf31a0167"} Nov 04 01:39:37.514873 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.514814 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s" event={"ID":"d0cbdf70-f15f-42bf-b3f0-929212551be1","Type":"ContainerDied","Data":"802677cfb932a79ce49c9b8a9a388bc2f463215a89a5d9d18bd16eea69dc1f0d"} Nov 04 01:39:37.514873 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.514836 5995 scope.go:122] "RemoveContainer" containerID="32ed93e35a841b8b2f673d534fcfc4f5514ce1bbf79435aa90f2256cf31a0167" Nov 04 01:39:37.525261 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.525241 5995 scope.go:122] "RemoveContainer" containerID="897c7b70436718d538c4e18c95e1e96c3fe13a02b5b0bb4196f6516823f89b87" Nov 04 01:39:37.535076 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.535053 5995 scope.go:122] "RemoveContainer" containerID="32ed93e35a841b8b2f673d534fcfc4f5514ce1bbf79435aa90f2256cf31a0167" Nov 04 01:39:37.535313 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:39:37.535293 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ed93e35a841b8b2f673d534fcfc4f5514ce1bbf79435aa90f2256cf31a0167\": container with ID starting with 32ed93e35a841b8b2f673d534fcfc4f5514ce1bbf79435aa90f2256cf31a0167 not found: ID does not exist" containerID="32ed93e35a841b8b2f673d534fcfc4f5514ce1bbf79435aa90f2256cf31a0167" Nov 04 01:39:37.535352 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.535323 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ed93e35a841b8b2f673d534fcfc4f5514ce1bbf79435aa90f2256cf31a0167"} err="failed to get container status \"32ed93e35a841b8b2f673d534fcfc4f5514ce1bbf79435aa90f2256cf31a0167\": rpc error: code = NotFound desc = could not find container \"32ed93e35a841b8b2f673d534fcfc4f5514ce1bbf79435aa90f2256cf31a0167\": container with ID starting with 32ed93e35a841b8b2f673d534fcfc4f5514ce1bbf79435aa90f2256cf31a0167 not found: ID does not exist" Nov 04 01:39:37.535352 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.535340 5995 scope.go:122] "RemoveContainer" containerID="897c7b70436718d538c4e18c95e1e96c3fe13a02b5b0bb4196f6516823f89b87" Nov 04 01:39:37.535579 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:39:37.535560 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897c7b70436718d538c4e18c95e1e96c3fe13a02b5b0bb4196f6516823f89b87\": container with ID starting with 897c7b70436718d538c4e18c95e1e96c3fe13a02b5b0bb4196f6516823f89b87 not found: ID does not exist" containerID="897c7b70436718d538c4e18c95e1e96c3fe13a02b5b0bb4196f6516823f89b87" Nov 04 01:39:37.535620 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.535586 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897c7b70436718d538c4e18c95e1e96c3fe13a02b5b0bb4196f6516823f89b87"} err="failed to get container status \"897c7b70436718d538c4e18c95e1e96c3fe13a02b5b0bb4196f6516823f89b87\": rpc error: code = NotFound desc = could not find container \"897c7b70436718d538c4e18c95e1e96c3fe13a02b5b0bb4196f6516823f89b87\": container with ID starting with 897c7b70436718d538c4e18c95e1e96c3fe13a02b5b0bb4196f6516823f89b87 not found: ID does not exist" Nov 04 01:39:37.670717 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.670681 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s"] Nov 04 01:39:37.676208 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:37.676184 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-6db2a7-openshift-default-77779b578f-smk2s"] Nov 04 01:39:38.129904 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.129874 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0cbdf70-f15f-42bf-b3f0-929212551be1" path="/var/lib/kubelet/pods/d0cbdf70-f15f-42bf-b3f0-929212551be1/volumes" Nov 04 01:39:38.606050 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.606017 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx"] Nov 04 01:39:38.606550 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.606534 5995 cpu_manager.go:414] "containerMap: removing container" podUID="d0cbdf70-f15f-42bf-b3f0-929212551be1" containerName="istio-proxy" Nov 04 01:39:38.606609 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.606551 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0cbdf70-f15f-42bf-b3f0-929212551be1" containerName="istio-proxy" Nov 04 01:39:38.606609 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.606561 5995 cpu_manager.go:414] "containerMap: removing container" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" Nov 04 01:39:38.606609 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.606566 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" Nov 04 01:39:38.606609 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.606590 5995 cpu_manager.go:414] "containerMap: removing container" podUID="d0cbdf70-f15f-42bf-b3f0-929212551be1" containerName="istio-proxy" Nov 04 01:39:38.606609 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.606596 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0cbdf70-f15f-42bf-b3f0-929212551be1" containerName="istio-proxy" Nov 04 01:39:38.606826 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.606692 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0cbdf70-f15f-42bf-b3f0-929212551be1" containerName="istio-proxy" Nov 04 01:39:38.606826 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.606702 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40ae805-9960-4a82-8ba6-b39b6bc6b458" containerName="payload-processing" Nov 04 01:39:38.610079 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.610063 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx" Nov 04 01:39:38.659292 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.659267 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:39:38.668588 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.668556 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6d1360-5222-47ab-83c8-8370d31dfda0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c348c19-ptvzx\" (UID: \"5a6d1360-5222-47ab-83c8-8370d31dfda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx" Nov 04 01:39:38.668729 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.668706 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwrt7\" (UniqueName: \"kubernetes.io/projected/5a6d1360-5222-47ab-83c8-8370d31dfda0-kube-api-access-hwrt7\") pod \"maas-api-revoke-sub-2c348c19-ptvzx\" (UID: \"5a6d1360-5222-47ab-83c8-8370d31dfda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx" Nov 04 01:39:38.668867 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.668847 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6d1360-5222-47ab-83c8-8370d31dfda0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c348c19-ptvzx\" (UID: \"5a6d1360-5222-47ab-83c8-8370d31dfda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx" Nov 04 01:39:38.709030 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.708992 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx"] Nov 04 01:39:38.724927 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.724898 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:39:38.725256 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.725071 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w799x\"" Nov 04 01:39:38.759432 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.759393 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:39:38.770297 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.770267 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwrt7\" (UniqueName: \"kubernetes.io/projected/5a6d1360-5222-47ab-83c8-8370d31dfda0-kube-api-access-hwrt7\") pod \"maas-api-revoke-sub-2c348c19-ptvzx\" (UID: \"5a6d1360-5222-47ab-83c8-8370d31dfda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx" Nov 04 01:39:38.770394 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.770350 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6d1360-5222-47ab-83c8-8370d31dfda0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c348c19-ptvzx\" (UID: \"5a6d1360-5222-47ab-83c8-8370d31dfda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx" Nov 04 01:39:38.770451 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.770408 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6d1360-5222-47ab-83c8-8370d31dfda0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c348c19-ptvzx\" (UID: \"5a6d1360-5222-47ab-83c8-8370d31dfda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx" Nov 04 01:39:38.771081 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.771064 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6d1360-5222-47ab-83c8-8370d31dfda0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c348c19-ptvzx\" (UID: \"5a6d1360-5222-47ab-83c8-8370d31dfda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx" Nov 04 01:39:38.772787 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.772765 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6d1360-5222-47ab-83c8-8370d31dfda0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c348c19-ptvzx\" (UID: \"5a6d1360-5222-47ab-83c8-8370d31dfda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx" Nov 04 01:39:38.827746 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.827720 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwrt7\" (UniqueName: \"kubernetes.io/projected/5a6d1360-5222-47ab-83c8-8370d31dfda0-kube-api-access-hwrt7\") pod \"maas-api-revoke-sub-2c348c19-ptvzx\" (UID: \"5a6d1360-5222-47ab-83c8-8370d31dfda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx" Nov 04 01:39:38.923472 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:38.923388 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx" Nov 04 01:39:39.295080 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:39.295046 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx"] Nov 04 01:39:39.300374 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:39:39.300346 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6d1360_5222_47ab_83c8_8370d31dfda0.slice/crio-4443aaf68120aa2ce4d145a4959ef7858e17241449a2a93ba6413ab62f2ee665 WatchSource:0}: Error finding container 4443aaf68120aa2ce4d145a4959ef7858e17241449a2a93ba6413ab62f2ee665: Status 404 returned error can't find the container with id 4443aaf68120aa2ce4d145a4959ef7858e17241449a2a93ba6413ab62f2ee665 Nov 04 01:39:39.531234 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:39.531201 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx" event={"ID":"5a6d1360-5222-47ab-83c8-8370d31dfda0","Type":"ContainerStarted","Data":"4443aaf68120aa2ce4d145a4959ef7858e17241449a2a93ba6413ab62f2ee665"} Nov 04 01:39:40.539340 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:40.539255 5995 generic.go:356] "Generic (PLEG): container finished" podID="5a6d1360-5222-47ab-83c8-8370d31dfda0" containerID="e2f42b1c891705a3d17094f081f6ccee2e9c7ed32ef3cc338a8aaa8801349224" exitCode=0 Nov 04 01:39:40.539700 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:40.539347 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx" event={"ID":"5a6d1360-5222-47ab-83c8-8370d31dfda0","Type":"ContainerDied","Data":"e2f42b1c891705a3d17094f081f6ccee2e9c7ed32ef3cc338a8aaa8801349224"} Nov 04 01:39:41.328848 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:41.328821 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" Nov 04 01:39:41.489308 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:41.489274 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz"] Nov 04 01:39:41.702114 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:41.702091 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx" Nov 04 01:39:41.819732 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:41.819699 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a6d1360-5222-47ab-83c8-8370d31dfda0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6d1360-5222-47ab-83c8-8370d31dfda0-maas-api-openshift-ca\") pod \"5a6d1360-5222-47ab-83c8-8370d31dfda0\" (UID: \"5a6d1360-5222-47ab-83c8-8370d31dfda0\") " Nov 04 01:39:41.819893 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:41.819828 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a6d1360-5222-47ab-83c8-8370d31dfda0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6d1360-5222-47ab-83c8-8370d31dfda0-maas-api-xks-ca\") pod \"5a6d1360-5222-47ab-83c8-8370d31dfda0\" (UID: \"5a6d1360-5222-47ab-83c8-8370d31dfda0\") " Nov 04 01:39:41.819973 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:41.819937 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a6d1360-5222-47ab-83c8-8370d31dfda0-kube-api-access-hwrt7\" (UniqueName: \"kubernetes.io/projected/5a6d1360-5222-47ab-83c8-8370d31dfda0-kube-api-access-hwrt7\") pod \"5a6d1360-5222-47ab-83c8-8370d31dfda0\" (UID: \"5a6d1360-5222-47ab-83c8-8370d31dfda0\") " Nov 04 01:39:41.820096 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:41.820073 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6d1360-5222-47ab-83c8-8370d31dfda0-maas-api-openshift-ca" pod "5a6d1360-5222-47ab-83c8-8370d31dfda0" (UID: "5a6d1360-5222-47ab-83c8-8370d31dfda0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:41.820430 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:41.820391 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6d1360-5222-47ab-83c8-8370d31dfda0-maas-api-openshift-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:41.822394 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:41.822372 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a6d1360-5222-47ab-83c8-8370d31dfda0-maas-api-xks-ca" pod "5a6d1360-5222-47ab-83c8-8370d31dfda0" (UID: "5a6d1360-5222-47ab-83c8-8370d31dfda0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:39:41.822521 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:41.822481 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6d1360-5222-47ab-83c8-8370d31dfda0-kube-api-access-hwrt7" pod "5a6d1360-5222-47ab-83c8-8370d31dfda0" (UID: "5a6d1360-5222-47ab-83c8-8370d31dfda0"). InnerVolumeSpecName "kube-api-access-hwrt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:41.921849 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:41.921779 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6d1360-5222-47ab-83c8-8370d31dfda0-maas-api-xks-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:41.921849 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:41.921805 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwrt7\" (UniqueName: \"kubernetes.io/projected/5a6d1360-5222-47ab-83c8-8370d31dfda0-kube-api-access-hwrt7\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:42.320247 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:42.320204 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r"] Nov 04 01:39:42.428171 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:42.428138 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82"] Nov 04 01:39:42.553045 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:42.553019 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx" Nov 04 01:39:42.553235 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:42.553055 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx" event={"ID":"5a6d1360-5222-47ab-83c8-8370d31dfda0","Type":"ContainerDied","Data":"4443aaf68120aa2ce4d145a4959ef7858e17241449a2a93ba6413ab62f2ee665"} Nov 04 01:39:42.553235 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:42.553098 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4443aaf68120aa2ce4d145a4959ef7858e17241449a2a93ba6413ab62f2ee665" Nov 04 01:39:46.490196 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:46.490154 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" podUID="7650b8ae-725e-470e-8960-5053aa977ea0" containerName="payload-processing" containerID="cri-o://7646d8e87b4446c538416301b0ef3e0f659653d3c2d75474390acd30f9214199" gracePeriod=25 Nov 04 01:39:46.824559 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:46.824527 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" Nov 04 01:39:46.832095 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:46.832066 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-kube-api-access-v72rq\" (UniqueName: \"kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-kube-api-access-v72rq\") pod \"7650b8ae-725e-470e-8960-5053aa977ea0\" (UID: \"7650b8ae-725e-470e-8960-5053aa977ea0\") " Nov 04 01:39:46.832263 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:46.832180 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7650b8ae-725e-470e-8960-5053aa977ea0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7650b8ae-725e-470e-8960-5053aa977ea0-plugins-config-volume\") pod \"7650b8ae-725e-470e-8960-5053aa977ea0\" (UID: \"7650b8ae-725e-470e-8960-5053aa977ea0\") " Nov 04 01:39:46.832322 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:46.832281 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7650b8ae-725e-470e-8960-5053aa977ea0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7650b8ae-725e-470e-8960-5053aa977ea0-routing-overlay-volume\") pod \"7650b8ae-725e-470e-8960-5053aa977ea0\" (UID: \"7650b8ae-725e-470e-8960-5053aa977ea0\") " Nov 04 01:39:46.832406 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:46.832385 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials\") pod \"7650b8ae-725e-470e-8960-5053aa977ea0\" (UID: \"7650b8ae-725e-470e-8960-5053aa977ea0\") " Nov 04 01:39:46.832641 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:46.832614 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7650b8ae-725e-470e-8960-5053aa977ea0-plugins-config-volume" pod "7650b8ae-725e-470e-8960-5053aa977ea0" (UID: "7650b8ae-725e-470e-8960-5053aa977ea0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:46.832641 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:46.832628 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7650b8ae-725e-470e-8960-5053aa977ea0-routing-overlay-volume" pod "7650b8ae-725e-470e-8960-5053aa977ea0" (UID: "7650b8ae-725e-470e-8960-5053aa977ea0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:46.832810 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:46.832794 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7650b8ae-725e-470e-8960-5053aa977ea0-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:46.832867 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:46.832815 5995 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7650b8ae-725e-470e-8960-5053aa977ea0-routing-overlay-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:46.834217 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:46.834194 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-kube-api-access-v72rq" pod "7650b8ae-725e-470e-8960-5053aa977ea0" (UID: "7650b8ae-725e-470e-8960-5053aa977ea0"). InnerVolumeSpecName "kube-api-access-v72rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:46.834381 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:46.834363 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials" pod "7650b8ae-725e-470e-8960-5053aa977ea0" (UID: "7650b8ae-725e-470e-8960-5053aa977ea0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:46.933618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:46.933580 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v72rq\" (UniqueName: \"kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-kube-api-access-v72rq\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:46.933618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:46.933610 5995 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7650b8ae-725e-470e-8960-5053aa977ea0-provider-credentials\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:47.320709 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.320649 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r" podUID="43e910c7-62fc-43a4-a513-847a2b968f65" containerName="payload-processing" containerID="cri-o://87ef638410a8cdd4cb8ddb63673b58caf0ddbfdd8a1b54fd04ad3036cb214982" gracePeriod=25 Nov 04 01:39:47.429381 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.429313 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82" podUID="54eca79d-c7bc-45b1-a815-119efc0f375e" containerName="payload-pre-processing" containerID="cri-o://e93b56ba94eac5614fe11d85fadc42b3cd0ab89ea836c713f53edcc792bafe1e" gracePeriod=25 Nov 04 01:39:47.570379 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.570355 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r" Nov 04 01:39:47.586624 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.586548 5995 generic.go:356] "Generic (PLEG): container finished" podID="54eca79d-c7bc-45b1-a815-119efc0f375e" containerID="e93b56ba94eac5614fe11d85fadc42b3cd0ab89ea836c713f53edcc792bafe1e" exitCode=0 Nov 04 01:39:47.586750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.586617 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82" event={"ID":"54eca79d-c7bc-45b1-a815-119efc0f375e","Type":"ContainerDied","Data":"e93b56ba94eac5614fe11d85fadc42b3cd0ab89ea836c713f53edcc792bafe1e"} Nov 04 01:39:47.587985 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.587960 5995 generic.go:356] "Generic (PLEG): container finished" podID="43e910c7-62fc-43a4-a513-847a2b968f65" containerID="87ef638410a8cdd4cb8ddb63673b58caf0ddbfdd8a1b54fd04ad3036cb214982" exitCode=0 Nov 04 01:39:47.588128 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.588011 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r" Nov 04 01:39:47.588128 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.588046 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r" event={"ID":"43e910c7-62fc-43a4-a513-847a2b968f65","Type":"ContainerDied","Data":"87ef638410a8cdd4cb8ddb63673b58caf0ddbfdd8a1b54fd04ad3036cb214982"} Nov 04 01:39:47.588128 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.588092 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r" event={"ID":"43e910c7-62fc-43a4-a513-847a2b968f65","Type":"ContainerDied","Data":"766ea1f6739ec18f0a03b8139cb6f24d25f147e34523bc5de495b8d811ef4fa3"} Nov 04 01:39:47.588128 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.588112 5995 scope.go:122] "RemoveContainer" containerID="87ef638410a8cdd4cb8ddb63673b58caf0ddbfdd8a1b54fd04ad3036cb214982" Nov 04 01:39:47.589618 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.589597 5995 generic.go:356] "Generic (PLEG): container finished" podID="7650b8ae-725e-470e-8960-5053aa977ea0" containerID="7646d8e87b4446c538416301b0ef3e0f659653d3c2d75474390acd30f9214199" exitCode=0 Nov 04 01:39:47.589725 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.589634 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" Nov 04 01:39:47.589725 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.589654 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" event={"ID":"7650b8ae-725e-470e-8960-5053aa977ea0","Type":"ContainerDied","Data":"7646d8e87b4446c538416301b0ef3e0f659653d3c2d75474390acd30f9214199"} Nov 04 01:39:47.589725 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.589691 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz" event={"ID":"7650b8ae-725e-470e-8960-5053aa977ea0","Type":"ContainerDied","Data":"6cfaaa86df14f466d64b24b986c257e0e9a32dc4395ea90e4e43ace1019fea23"} Nov 04 01:39:47.603887 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.603768 5995 scope.go:122] "RemoveContainer" containerID="87ef638410a8cdd4cb8ddb63673b58caf0ddbfdd8a1b54fd04ad3036cb214982" Nov 04 01:39:47.604959 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:39:47.604793 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ef638410a8cdd4cb8ddb63673b58caf0ddbfdd8a1b54fd04ad3036cb214982\": container with ID starting with 87ef638410a8cdd4cb8ddb63673b58caf0ddbfdd8a1b54fd04ad3036cb214982 not found: ID does not exist" containerID="87ef638410a8cdd4cb8ddb63673b58caf0ddbfdd8a1b54fd04ad3036cb214982" Nov 04 01:39:47.604959 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.604835 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ef638410a8cdd4cb8ddb63673b58caf0ddbfdd8a1b54fd04ad3036cb214982"} err="failed to get container status \"87ef638410a8cdd4cb8ddb63673b58caf0ddbfdd8a1b54fd04ad3036cb214982\": rpc error: code = NotFound desc = could not find container \"87ef638410a8cdd4cb8ddb63673b58caf0ddbfdd8a1b54fd04ad3036cb214982\": container with ID starting with 87ef638410a8cdd4cb8ddb63673b58caf0ddbfdd8a1b54fd04ad3036cb214982 not found: ID does not exist" Nov 04 01:39:47.604959 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.604861 5995 scope.go:122] "RemoveContainer" containerID="7646d8e87b4446c538416301b0ef3e0f659653d3c2d75474390acd30f9214199" Nov 04 01:39:47.618446 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.618427 5995 scope.go:122] "RemoveContainer" containerID="7646d8e87b4446c538416301b0ef3e0f659653d3c2d75474390acd30f9214199" Nov 04 01:39:47.618759 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:39:47.618741 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7646d8e87b4446c538416301b0ef3e0f659653d3c2d75474390acd30f9214199\": container with ID starting with 7646d8e87b4446c538416301b0ef3e0f659653d3c2d75474390acd30f9214199 not found: ID does not exist" containerID="7646d8e87b4446c538416301b0ef3e0f659653d3c2d75474390acd30f9214199" Nov 04 01:39:47.618821 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.618777 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7646d8e87b4446c538416301b0ef3e0f659653d3c2d75474390acd30f9214199"} err="failed to get container status \"7646d8e87b4446c538416301b0ef3e0f659653d3c2d75474390acd30f9214199\": rpc error: code = NotFound desc = could not find container \"7646d8e87b4446c538416301b0ef3e0f659653d3c2d75474390acd30f9214199\": container with ID starting with 7646d8e87b4446c538416301b0ef3e0f659653d3c2d75474390acd30f9214199 not found: ID does not exist" Nov 04 01:39:47.744027 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.743998 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43e910c7-62fc-43a4-a513-847a2b968f65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e910c7-62fc-43a4-a513-847a2b968f65-plugins-config-volume\") pod \"43e910c7-62fc-43a4-a513-847a2b968f65\" (UID: \"43e910c7-62fc-43a4-a513-847a2b968f65\") " Nov 04 01:39:47.744195 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.744052 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e910c7-62fc-43a4-a513-847a2b968f65-kube-api-access-stvts\" (UniqueName: \"kubernetes.io/projected/43e910c7-62fc-43a4-a513-847a2b968f65-kube-api-access-stvts\") pod \"43e910c7-62fc-43a4-a513-847a2b968f65\" (UID: \"43e910c7-62fc-43a4-a513-847a2b968f65\") " Nov 04 01:39:47.744384 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.744359 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e910c7-62fc-43a4-a513-847a2b968f65-plugins-config-volume" pod "43e910c7-62fc-43a4-a513-847a2b968f65" (UID: "43e910c7-62fc-43a4-a513-847a2b968f65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:47.746254 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.746232 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e910c7-62fc-43a4-a513-847a2b968f65-kube-api-access-stvts" pod "43e910c7-62fc-43a4-a513-847a2b968f65" (UID: "43e910c7-62fc-43a4-a513-847a2b968f65"). InnerVolumeSpecName "kube-api-access-stvts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:47.771179 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.771155 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82" Nov 04 01:39:47.799749 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.799710 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz"] Nov 04 01:39:47.821967 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.821936 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6fddb87f65-hxqhz"] Nov 04 01:39:47.845130 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.845048 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e910c7-62fc-43a4-a513-847a2b968f65-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:47.845130 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.845075 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stvts\" (UniqueName: \"kubernetes.io/projected/43e910c7-62fc-43a4-a513-847a2b968f65-kube-api-access-stvts\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:47.945711 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.945679 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54eca79d-c7bc-45b1-a815-119efc0f375e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54eca79d-c7bc-45b1-a815-119efc0f375e-plugins-config-volume\") pod \"54eca79d-c7bc-45b1-a815-119efc0f375e\" (UID: \"54eca79d-c7bc-45b1-a815-119efc0f375e\") " Nov 04 01:39:47.945908 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.945807 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54eca79d-c7bc-45b1-a815-119efc0f375e-kube-api-access-pbzwn\" (UniqueName: \"kubernetes.io/projected/54eca79d-c7bc-45b1-a815-119efc0f375e-kube-api-access-pbzwn\") pod \"54eca79d-c7bc-45b1-a815-119efc0f375e\" (UID: \"54eca79d-c7bc-45b1-a815-119efc0f375e\") " Nov 04 01:39:47.946112 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.946088 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54eca79d-c7bc-45b1-a815-119efc0f375e-plugins-config-volume" pod "54eca79d-c7bc-45b1-a815-119efc0f375e" (UID: "54eca79d-c7bc-45b1-a815-119efc0f375e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:47.947951 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:47.947918 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54eca79d-c7bc-45b1-a815-119efc0f375e-kube-api-access-pbzwn" pod "54eca79d-c7bc-45b1-a815-119efc0f375e" (UID: "54eca79d-c7bc-45b1-a815-119efc0f375e"). InnerVolumeSpecName "kube-api-access-pbzwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:48.048507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:48.047768 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54eca79d-c7bc-45b1-a815-119efc0f375e-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:48.048507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:48.047816 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbzwn\" (UniqueName: \"kubernetes.io/projected/54eca79d-c7bc-45b1-a815-119efc0f375e-kube-api-access-pbzwn\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:48.098877 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:48.098787 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r"] Nov 04 01:39:48.103503 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:48.103475 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d057d7-687bd5c98d-n7m6r"] Nov 04 01:39:48.129507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:48.129475 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43e910c7-62fc-43a4-a513-847a2b968f65" path="/var/lib/kubelet/pods/43e910c7-62fc-43a4-a513-847a2b968f65/volumes" Nov 04 01:39:48.129934 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:48.129919 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7650b8ae-725e-470e-8960-5053aa977ea0" path="/var/lib/kubelet/pods/7650b8ae-725e-470e-8960-5053aa977ea0/volumes" Nov 04 01:39:48.596922 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:48.596890 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82" Nov 04 01:39:48.597387 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:48.596899 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82" event={"ID":"54eca79d-c7bc-45b1-a815-119efc0f375e","Type":"ContainerDied","Data":"f14f00e6b47d7a72838a7d1a3d09ab5299e317b3611eb9d1885110069f5776c1"} Nov 04 01:39:48.597387 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:48.597034 5995 scope.go:122] "RemoveContainer" containerID="e93b56ba94eac5614fe11d85fadc42b3cd0ab89ea836c713f53edcc792bafe1e" Nov 04 01:39:48.752794 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:48.752757 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82"] Nov 04 01:39:48.799474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:48.799354 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d057d7-7678c7db67-k5x82"] Nov 04 01:39:49.641154 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:39:49.641118 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:39:50.128317 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:50.128285 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54eca79d-c7bc-45b1-a815-119efc0f375e" path="/var/lib/kubelet/pods/54eca79d-c7bc-45b1-a815-119efc0f375e/volumes" Nov 04 01:39:52.351957 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:52.351912 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:39:53.167078 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.167042 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-765779c7c6-wn22v"] Nov 04 01:39:53.167569 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.167551 5995 cpu_manager.go:414] "containerMap: removing container" podUID="43e910c7-62fc-43a4-a513-847a2b968f65" containerName="payload-processing" Nov 04 01:39:53.167569 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.167568 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e910c7-62fc-43a4-a513-847a2b968f65" containerName="payload-processing" Nov 04 01:39:53.167703 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.167583 5995 cpu_manager.go:414] "containerMap: removing container" podUID="5a6d1360-5222-47ab-83c8-8370d31dfda0" containerName="revoke-keys" Nov 04 01:39:53.167703 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.167591 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a6d1360-5222-47ab-83c8-8370d31dfda0" containerName="revoke-keys" Nov 04 01:39:53.167703 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.167613 5995 cpu_manager.go:414] "containerMap: removing container" podUID="7650b8ae-725e-470e-8960-5053aa977ea0" containerName="payload-processing" Nov 04 01:39:53.167703 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.167619 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7650b8ae-725e-470e-8960-5053aa977ea0" containerName="payload-processing" Nov 04 01:39:53.167703 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.167655 5995 cpu_manager.go:414] "containerMap: removing container" podUID="54eca79d-c7bc-45b1-a815-119efc0f375e" containerName="payload-pre-processing" Nov 04 01:39:53.167703 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.167662 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54eca79d-c7bc-45b1-a815-119efc0f375e" containerName="payload-pre-processing" Nov 04 01:39:53.167906 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.167833 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0cbdf70-f15f-42bf-b3f0-929212551be1" containerName="istio-proxy" Nov 04 01:39:53.167906 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.167850 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="54eca79d-c7bc-45b1-a815-119efc0f375e" containerName="payload-pre-processing" Nov 04 01:39:53.167906 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.167876 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e910c7-62fc-43a4-a513-847a2b968f65" containerName="payload-processing" Nov 04 01:39:53.167906 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.167887 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a6d1360-5222-47ab-83c8-8370d31dfda0" containerName="revoke-keys" Nov 04 01:39:53.167906 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.167898 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="7650b8ae-725e-470e-8960-5053aa977ea0" containerName="payload-processing" Nov 04 01:39:53.173071 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.173053 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" Nov 04 01:39:53.272965 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.272933 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-765779c7c6-wn22v"] Nov 04 01:39:53.316680 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.316648 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fskm6\" (UniqueName: \"kubernetes.io/projected/4cfdc4da-e36c-43f5-9491-cca19d01a097-kube-api-access-fskm6\") pod \"payload-processing-external-model-765779c7c6-wn22v\" (UID: \"4cfdc4da-e36c-43f5-9491-cca19d01a097\") " pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" Nov 04 01:39:53.316859 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.316716 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4cfdc4da-e36c-43f5-9491-cca19d01a097-provider-credentials\") pod \"payload-processing-external-model-765779c7c6-wn22v\" (UID: \"4cfdc4da-e36c-43f5-9491-cca19d01a097\") " pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" Nov 04 01:39:53.316859 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.316811 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cfdc4da-e36c-43f5-9491-cca19d01a097-plugins-config-volume\") pod \"payload-processing-external-model-765779c7c6-wn22v\" (UID: \"4cfdc4da-e36c-43f5-9491-cca19d01a097\") " pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" Nov 04 01:39:53.316859 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.316839 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4cfdc4da-e36c-43f5-9491-cca19d01a097-routing-overlay-volume\") pod \"payload-processing-external-model-765779c7c6-wn22v\" (UID: \"4cfdc4da-e36c-43f5-9491-cca19d01a097\") " pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" Nov 04 01:39:53.417939 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.417865 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4cfdc4da-e36c-43f5-9491-cca19d01a097-provider-credentials\") pod \"payload-processing-external-model-765779c7c6-wn22v\" (UID: \"4cfdc4da-e36c-43f5-9491-cca19d01a097\") " pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" Nov 04 01:39:53.418407 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.417944 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cfdc4da-e36c-43f5-9491-cca19d01a097-plugins-config-volume\") pod \"payload-processing-external-model-765779c7c6-wn22v\" (UID: \"4cfdc4da-e36c-43f5-9491-cca19d01a097\") " pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" Nov 04 01:39:53.418407 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.417976 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4cfdc4da-e36c-43f5-9491-cca19d01a097-routing-overlay-volume\") pod \"payload-processing-external-model-765779c7c6-wn22v\" (UID: \"4cfdc4da-e36c-43f5-9491-cca19d01a097\") " pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" Nov 04 01:39:53.418407 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.418074 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fskm6\" (UniqueName: \"kubernetes.io/projected/4cfdc4da-e36c-43f5-9491-cca19d01a097-kube-api-access-fskm6\") pod \"payload-processing-external-model-765779c7c6-wn22v\" (UID: \"4cfdc4da-e36c-43f5-9491-cca19d01a097\") " pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" Nov 04 01:39:53.418874 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.418844 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4cfdc4da-e36c-43f5-9491-cca19d01a097-routing-overlay-volume\") pod \"payload-processing-external-model-765779c7c6-wn22v\" (UID: \"4cfdc4da-e36c-43f5-9491-cca19d01a097\") " pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" Nov 04 01:39:53.419036 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.419021 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cfdc4da-e36c-43f5-9491-cca19d01a097-plugins-config-volume\") pod \"payload-processing-external-model-765779c7c6-wn22v\" (UID: \"4cfdc4da-e36c-43f5-9491-cca19d01a097\") " pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" Nov 04 01:39:53.421326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.421296 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4cfdc4da-e36c-43f5-9491-cca19d01a097-provider-credentials\") pod \"payload-processing-external-model-765779c7c6-wn22v\" (UID: \"4cfdc4da-e36c-43f5-9491-cca19d01a097\") " pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" Nov 04 01:39:53.475602 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.475569 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fskm6\" (UniqueName: \"kubernetes.io/projected/4cfdc4da-e36c-43f5-9491-cca19d01a097-kube-api-access-fskm6\") pod \"payload-processing-external-model-765779c7c6-wn22v\" (UID: \"4cfdc4da-e36c-43f5-9491-cca19d01a097\") " pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" Nov 04 01:39:53.486502 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.486470 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" Nov 04 01:39:53.892547 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:53.892513 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-765779c7c6-wn22v"] Nov 04 01:39:53.898211 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:39:53.898180 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cfdc4da_e36c_43f5_9491_cca19d01a097.slice/crio-8a1e01ab8a58a4bf775587353f33e6064549ae4bd00f1b6c11ee3b03cd4a9313 WatchSource:0}: Error finding container 8a1e01ab8a58a4bf775587353f33e6064549ae4bd00f1b6c11ee3b03cd4a9313: Status 404 returned error can't find the container with id 8a1e01ab8a58a4bf775587353f33e6064549ae4bd00f1b6c11ee3b03cd4a9313 Nov 04 01:39:54.641616 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.641575 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" event={"ID":"4cfdc4da-e36c-43f5-9491-cca19d01a097","Type":"ContainerStarted","Data":"8dc4e1fdbaded35e27deab16d1a6d39f1efd145ded6b900221727e104270ed78"} Nov 04 01:39:54.642060 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.641640 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" event={"ID":"4cfdc4da-e36c-43f5-9491-cca19d01a097","Type":"ContainerStarted","Data":"8a1e01ab8a58a4bf775587353f33e6064549ae4bd00f1b6c11ee3b03cd4a9313"} Nov 04 01:39:54.642060 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.641666 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" Nov 04 01:39:54.719145 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.719104 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj"] Nov 04 01:39:54.723334 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.723305 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj" Nov 04 01:39:54.752709 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.752663 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podStartSLOduration=1.7526478220000001 podStartE2EDuration="1.752647822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:39:54.750892211 +0000 UTC m=+1744.782625601" watchObservedRunningTime="2026-11-04 01:39:54.752647822 +0000 UTC m=+1744.784381211" Nov 04 01:39:54.775110 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.775073 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:39:54.775330 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.775165 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w799x\"" Nov 04 01:39:54.775330 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.775191 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:39:54.778994 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.778780 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj"] Nov 04 01:39:54.797496 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.797470 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:39:54.832653 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.832619 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acb3152-ac36-4c7c-8f76-84fb19118a7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-908e8a77-7t6gj\" (UID: \"4acb3152-ac36-4c7c-8f76-84fb19118a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj" Nov 04 01:39:54.832797 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.832745 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn7mm\" (UniqueName: \"kubernetes.io/projected/4acb3152-ac36-4c7c-8f76-84fb19118a7c-kube-api-access-mn7mm\") pod \"maas-api-revoke-sub-908e8a77-7t6gj\" (UID: \"4acb3152-ac36-4c7c-8f76-84fb19118a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj" Nov 04 01:39:54.832833 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.832794 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acb3152-ac36-4c7c-8f76-84fb19118a7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-908e8a77-7t6gj\" (UID: \"4acb3152-ac36-4c7c-8f76-84fb19118a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj" Nov 04 01:39:54.934004 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.933907 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acb3152-ac36-4c7c-8f76-84fb19118a7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-908e8a77-7t6gj\" (UID: \"4acb3152-ac36-4c7c-8f76-84fb19118a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj" Nov 04 01:39:54.934189 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.934011 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7mm\" (UniqueName: \"kubernetes.io/projected/4acb3152-ac36-4c7c-8f76-84fb19118a7c-kube-api-access-mn7mm\") pod \"maas-api-revoke-sub-908e8a77-7t6gj\" (UID: \"4acb3152-ac36-4c7c-8f76-84fb19118a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj" Nov 04 01:39:54.934189 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.934062 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acb3152-ac36-4c7c-8f76-84fb19118a7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-908e8a77-7t6gj\" (UID: \"4acb3152-ac36-4c7c-8f76-84fb19118a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj" Nov 04 01:39:54.934616 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.934594 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acb3152-ac36-4c7c-8f76-84fb19118a7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-908e8a77-7t6gj\" (UID: \"4acb3152-ac36-4c7c-8f76-84fb19118a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj" Nov 04 01:39:54.936435 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.936401 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acb3152-ac36-4c7c-8f76-84fb19118a7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-908e8a77-7t6gj\" (UID: \"4acb3152-ac36-4c7c-8f76-84fb19118a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj" Nov 04 01:39:54.991790 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:54.991753 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn7mm\" (UniqueName: \"kubernetes.io/projected/4acb3152-ac36-4c7c-8f76-84fb19118a7c-kube-api-access-mn7mm\") pod \"maas-api-revoke-sub-908e8a77-7t6gj\" (UID: \"4acb3152-ac36-4c7c-8f76-84fb19118a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj" Nov 04 01:39:55.037542 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:55.037512 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj" Nov 04 01:39:55.415481 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:55.415446 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj"] Nov 04 01:39:55.420256 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:39:55.420228 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4acb3152_ac36_4c7c_8f76_84fb19118a7c.slice/crio-22f6eb2c867e70a918df02c64352e1cbdb55f5ff3ebd9c748418c545881c0c83 WatchSource:0}: Error finding container 22f6eb2c867e70a918df02c64352e1cbdb55f5ff3ebd9c748418c545881c0c83: Status 404 returned error can't find the container with id 22f6eb2c867e70a918df02c64352e1cbdb55f5ff3ebd9c748418c545881c0c83 Nov 04 01:39:55.652374 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:55.652337 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj" event={"ID":"4acb3152-ac36-4c7c-8f76-84fb19118a7c","Type":"ContainerStarted","Data":"22f6eb2c867e70a918df02c64352e1cbdb55f5ff3ebd9c748418c545881c0c83"} Nov 04 01:39:56.660201 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:56.660163 5995 generic.go:356] "Generic (PLEG): container finished" podID="4acb3152-ac36-4c7c-8f76-84fb19118a7c" containerID="80917cac84590f617638880741ab8638d5eb394aae54f71c85e7f5e0718ed57a" exitCode=0 Nov 04 01:39:56.660634 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:56.660200 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj" event={"ID":"4acb3152-ac36-4c7c-8f76-84fb19118a7c","Type":"ContainerDied","Data":"80917cac84590f617638880741ab8638d5eb394aae54f71c85e7f5e0718ed57a"} Nov 04 01:39:56.831103 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:56.831068 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp"] Nov 04 01:39:56.835204 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:56.835182 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp" Nov 04 01:39:56.912454 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:56.907790 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp"] Nov 04 01:39:56.956112 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:56.956081 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7pnb\" (UniqueName: \"kubernetes.io/projected/0514c9e0-8b44-42a4-9d14-561aae8f2d05-kube-api-access-m7pnb\") pod \"maas-api-revoke-sub-37193fb8-rzvdp\" (UID: \"0514c9e0-8b44-42a4-9d14-561aae8f2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp" Nov 04 01:39:56.956281 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:56.956130 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0514c9e0-8b44-42a4-9d14-561aae8f2d05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37193fb8-rzvdp\" (UID: \"0514c9e0-8b44-42a4-9d14-561aae8f2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp" Nov 04 01:39:56.956344 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:56.956294 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0514c9e0-8b44-42a4-9d14-561aae8f2d05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37193fb8-rzvdp\" (UID: \"0514c9e0-8b44-42a4-9d14-561aae8f2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp" Nov 04 01:39:57.057020 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:57.056986 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7pnb\" (UniqueName: \"kubernetes.io/projected/0514c9e0-8b44-42a4-9d14-561aae8f2d05-kube-api-access-m7pnb\") pod \"maas-api-revoke-sub-37193fb8-rzvdp\" (UID: \"0514c9e0-8b44-42a4-9d14-561aae8f2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp" Nov 04 01:39:57.057204 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:57.057039 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0514c9e0-8b44-42a4-9d14-561aae8f2d05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37193fb8-rzvdp\" (UID: \"0514c9e0-8b44-42a4-9d14-561aae8f2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp" Nov 04 01:39:57.057204 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:57.057103 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0514c9e0-8b44-42a4-9d14-561aae8f2d05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37193fb8-rzvdp\" (UID: \"0514c9e0-8b44-42a4-9d14-561aae8f2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp" Nov 04 01:39:57.057739 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:57.057719 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0514c9e0-8b44-42a4-9d14-561aae8f2d05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37193fb8-rzvdp\" (UID: \"0514c9e0-8b44-42a4-9d14-561aae8f2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp" Nov 04 01:39:57.059570 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:57.059542 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0514c9e0-8b44-42a4-9d14-561aae8f2d05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37193fb8-rzvdp\" (UID: \"0514c9e0-8b44-42a4-9d14-561aae8f2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp" Nov 04 01:39:57.115027 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:57.114949 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7pnb\" (UniqueName: \"kubernetes.io/projected/0514c9e0-8b44-42a4-9d14-561aae8f2d05-kube-api-access-m7pnb\") pod \"maas-api-revoke-sub-37193fb8-rzvdp\" (UID: \"0514c9e0-8b44-42a4-9d14-561aae8f2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp" Nov 04 01:39:57.148055 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:57.148026 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp" Nov 04 01:39:57.525000 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:57.524956 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp"] Nov 04 01:39:57.530642 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:39:57.530615 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0514c9e0_8b44_42a4_9d14_561aae8f2d05.slice/crio-ada80a8d0d334fd8a7846cbf2b6f8ebab43c70bb141a72d2c46b0c0c8e0ba684 WatchSource:0}: Error finding container ada80a8d0d334fd8a7846cbf2b6f8ebab43c70bb141a72d2c46b0c0c8e0ba684: Status 404 returned error can't find the container with id ada80a8d0d334fd8a7846cbf2b6f8ebab43c70bb141a72d2c46b0c0c8e0ba684 Nov 04 01:39:57.668246 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:57.668212 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp" event={"ID":"0514c9e0-8b44-42a4-9d14-561aae8f2d05","Type":"ContainerStarted","Data":"ada80a8d0d334fd8a7846cbf2b6f8ebab43c70bb141a72d2c46b0c0c8e0ba684"} Nov 04 01:39:57.820798 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:57.820762 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj" Nov 04 01:39:57.966858 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:57.966818 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4acb3152-ac36-4c7c-8f76-84fb19118a7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acb3152-ac36-4c7c-8f76-84fb19118a7c-maas-api-openshift-ca\") pod \"4acb3152-ac36-4c7c-8f76-84fb19118a7c\" (UID: \"4acb3152-ac36-4c7c-8f76-84fb19118a7c\") " Nov 04 01:39:57.967052 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:57.966906 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4acb3152-ac36-4c7c-8f76-84fb19118a7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acb3152-ac36-4c7c-8f76-84fb19118a7c-maas-api-xks-ca\") pod \"4acb3152-ac36-4c7c-8f76-84fb19118a7c\" (UID: \"4acb3152-ac36-4c7c-8f76-84fb19118a7c\") " Nov 04 01:39:57.967052 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:57.966939 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4acb3152-ac36-4c7c-8f76-84fb19118a7c-kube-api-access-mn7mm\" (UniqueName: \"kubernetes.io/projected/4acb3152-ac36-4c7c-8f76-84fb19118a7c-kube-api-access-mn7mm\") pod \"4acb3152-ac36-4c7c-8f76-84fb19118a7c\" (UID: \"4acb3152-ac36-4c7c-8f76-84fb19118a7c\") " Nov 04 01:39:57.967246 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:57.967223 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4acb3152-ac36-4c7c-8f76-84fb19118a7c-maas-api-openshift-ca" pod "4acb3152-ac36-4c7c-8f76-84fb19118a7c" (UID: "4acb3152-ac36-4c7c-8f76-84fb19118a7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:57.967354 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:57.967332 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acb3152-ac36-4c7c-8f76-84fb19118a7c-maas-api-openshift-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:57.969433 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:57.969393 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4acb3152-ac36-4c7c-8f76-84fb19118a7c-maas-api-xks-ca" pod "4acb3152-ac36-4c7c-8f76-84fb19118a7c" (UID: "4acb3152-ac36-4c7c-8f76-84fb19118a7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:39:57.969534 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:57.969517 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4acb3152-ac36-4c7c-8f76-84fb19118a7c-kube-api-access-mn7mm" pod "4acb3152-ac36-4c7c-8f76-84fb19118a7c" (UID: "4acb3152-ac36-4c7c-8f76-84fb19118a7c"). InnerVolumeSpecName "kube-api-access-mn7mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:58.068013 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:58.067931 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acb3152-ac36-4c7c-8f76-84fb19118a7c-maas-api-xks-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:58.068013 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:58.067960 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn7mm\" (UniqueName: \"kubernetes.io/projected/4acb3152-ac36-4c7c-8f76-84fb19118a7c-kube-api-access-mn7mm\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:58.677145 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:58.677106 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj" Nov 04 01:39:58.677145 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:58.677135 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj" event={"ID":"4acb3152-ac36-4c7c-8f76-84fb19118a7c","Type":"ContainerDied","Data":"22f6eb2c867e70a918df02c64352e1cbdb55f5ff3ebd9c748418c545881c0c83"} Nov 04 01:39:58.677696 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:58.677193 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22f6eb2c867e70a918df02c64352e1cbdb55f5ff3ebd9c748418c545881c0c83" Nov 04 01:39:58.678745 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:58.678715 5995 generic.go:356] "Generic (PLEG): container finished" podID="0514c9e0-8b44-42a4-9d14-561aae8f2d05" containerID="632dd861ed0aa4150a6f5ea3b4b3e92fc3e59091bbaa1ff8d1a4249ba337be23" exitCode=0 Nov 04 01:39:58.678867 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:58.678787 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp" event={"ID":"0514c9e0-8b44-42a4-9d14-561aae8f2d05","Type":"ContainerDied","Data":"632dd861ed0aa4150a6f5ea3b4b3e92fc3e59091bbaa1ff8d1a4249ba337be23"} Nov 04 01:39:59.055695 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:59.055659 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g"] Nov 04 01:39:59.832864 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:59.832838 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp" Nov 04 01:39:59.987950 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:59.987922 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0514c9e0-8b44-42a4-9d14-561aae8f2d05-kube-api-access-m7pnb\" (UniqueName: \"kubernetes.io/projected/0514c9e0-8b44-42a4-9d14-561aae8f2d05-kube-api-access-m7pnb\") pod \"0514c9e0-8b44-42a4-9d14-561aae8f2d05\" (UID: \"0514c9e0-8b44-42a4-9d14-561aae8f2d05\") " Nov 04 01:39:59.988113 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:59.987966 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0514c9e0-8b44-42a4-9d14-561aae8f2d05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0514c9e0-8b44-42a4-9d14-561aae8f2d05-maas-api-xks-ca\") pod \"0514c9e0-8b44-42a4-9d14-561aae8f2d05\" (UID: \"0514c9e0-8b44-42a4-9d14-561aae8f2d05\") " Nov 04 01:39:59.988113 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:59.988062 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0514c9e0-8b44-42a4-9d14-561aae8f2d05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0514c9e0-8b44-42a4-9d14-561aae8f2d05-maas-api-openshift-ca\") pod \"0514c9e0-8b44-42a4-9d14-561aae8f2d05\" (UID: \"0514c9e0-8b44-42a4-9d14-561aae8f2d05\") " Nov 04 01:39:59.988541 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:59.988515 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0514c9e0-8b44-42a4-9d14-561aae8f2d05-maas-api-openshift-ca" pod "0514c9e0-8b44-42a4-9d14-561aae8f2d05" (UID: "0514c9e0-8b44-42a4-9d14-561aae8f2d05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:59.990165 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:59.990139 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0514c9e0-8b44-42a4-9d14-561aae8f2d05-maas-api-xks-ca" pod "0514c9e0-8b44-42a4-9d14-561aae8f2d05" (UID: "0514c9e0-8b44-42a4-9d14-561aae8f2d05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:39:59.990273 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:39:59.990182 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0514c9e0-8b44-42a4-9d14-561aae8f2d05-kube-api-access-m7pnb" pod "0514c9e0-8b44-42a4-9d14-561aae8f2d05" (UID: "0514c9e0-8b44-42a4-9d14-561aae8f2d05"). InnerVolumeSpecName "kube-api-access-m7pnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:00.089042 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:00.089005 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7pnb\" (UniqueName: \"kubernetes.io/projected/0514c9e0-8b44-42a4-9d14-561aae8f2d05-kube-api-access-m7pnb\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:00.089042 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:00.089040 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0514c9e0-8b44-42a4-9d14-561aae8f2d05-maas-api-xks-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:00.089222 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:00.089055 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0514c9e0-8b44-42a4-9d14-561aae8f2d05-maas-api-openshift-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:00.692737 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:00.692704 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp" Nov 04 01:40:00.692909 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:00.692737 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp" event={"ID":"0514c9e0-8b44-42a4-9d14-561aae8f2d05","Type":"ContainerDied","Data":"ada80a8d0d334fd8a7846cbf2b6f8ebab43c70bb141a72d2c46b0c0c8e0ba684"} Nov 04 01:40:00.692909 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:00.692813 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ada80a8d0d334fd8a7846cbf2b6f8ebab43c70bb141a72d2c46b0c0c8e0ba684" Nov 04 01:40:01.166450 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:01.166362 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v"] Nov 04 01:40:05.655767 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:05.655733 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:40:10.488706 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.488659 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5"] Nov 04 01:40:10.489734 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.489708 5995 cpu_manager.go:414] "containerMap: removing container" podUID="4acb3152-ac36-4c7c-8f76-84fb19118a7c" containerName="revoke-keys" Nov 04 01:40:10.489734 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.489733 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4acb3152-ac36-4c7c-8f76-84fb19118a7c" containerName="revoke-keys" Nov 04 01:40:10.489914 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.489781 5995 cpu_manager.go:414] "containerMap: removing container" podUID="0514c9e0-8b44-42a4-9d14-561aae8f2d05" containerName="revoke-keys" Nov 04 01:40:10.489914 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.489789 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0514c9e0-8b44-42a4-9d14-561aae8f2d05" containerName="revoke-keys" Nov 04 01:40:10.490070 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.490055 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="4acb3152-ac36-4c7c-8f76-84fb19118a7c" containerName="revoke-keys" Nov 04 01:40:10.490116 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.490082 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="0514c9e0-8b44-42a4-9d14-561aae8f2d05" containerName="revoke-keys" Nov 04 01:40:10.496178 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.496154 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5" Nov 04 01:40:10.545363 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.545318 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5"] Nov 04 01:40:10.588348 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.588313 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:40:10.588348 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.588312 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:40:10.588608 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.588460 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:40:10.591870 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.591851 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w799x\"" Nov 04 01:40:10.599578 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.599548 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3934445a-0626-428a-a390-a49783fc7e73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90515613-r97r5\" (UID: \"3934445a-0626-428a-a390-a49783fc7e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5" Nov 04 01:40:10.599747 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.599731 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvnnd\" (UniqueName: \"kubernetes.io/projected/3934445a-0626-428a-a390-a49783fc7e73-kube-api-access-nvnnd\") pod \"maas-api-revoke-sub-90515613-r97r5\" (UID: \"3934445a-0626-428a-a390-a49783fc7e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5" Nov 04 01:40:10.599808 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.599769 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3934445a-0626-428a-a390-a49783fc7e73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90515613-r97r5\" (UID: \"3934445a-0626-428a-a390-a49783fc7e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5" Nov 04 01:40:10.700738 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.700701 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvnnd\" (UniqueName: \"kubernetes.io/projected/3934445a-0626-428a-a390-a49783fc7e73-kube-api-access-nvnnd\") pod \"maas-api-revoke-sub-90515613-r97r5\" (UID: \"3934445a-0626-428a-a390-a49783fc7e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5" Nov 04 01:40:10.700926 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.700753 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3934445a-0626-428a-a390-a49783fc7e73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90515613-r97r5\" (UID: \"3934445a-0626-428a-a390-a49783fc7e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5" Nov 04 01:40:10.700926 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.700841 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3934445a-0626-428a-a390-a49783fc7e73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90515613-r97r5\" (UID: \"3934445a-0626-428a-a390-a49783fc7e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5" Nov 04 01:40:10.701617 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.701590 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3934445a-0626-428a-a390-a49783fc7e73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90515613-r97r5\" (UID: \"3934445a-0626-428a-a390-a49783fc7e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5" Nov 04 01:40:10.703451 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.703406 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3934445a-0626-428a-a390-a49783fc7e73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90515613-r97r5\" (UID: \"3934445a-0626-428a-a390-a49783fc7e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5" Nov 04 01:40:10.758829 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.758753 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvnnd\" (UniqueName: \"kubernetes.io/projected/3934445a-0626-428a-a390-a49783fc7e73-kube-api-access-nvnnd\") pod \"maas-api-revoke-sub-90515613-r97r5\" (UID: \"3934445a-0626-428a-a390-a49783fc7e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5" Nov 04 01:40:10.810019 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:10.809981 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5" Nov 04 01:40:11.184406 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:11.184330 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5"] Nov 04 01:40:11.188519 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:40:11.188488 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3934445a_0626_428a_a390_a49783fc7e73.slice/crio-2bee56fcbad4222f95ef940a393e36891e30a7305e3d50cdfb415fe5edc3d22f WatchSource:0}: Error finding container 2bee56fcbad4222f95ef940a393e36891e30a7305e3d50cdfb415fe5edc3d22f: Status 404 returned error can't find the container with id 2bee56fcbad4222f95ef940a393e36891e30a7305e3d50cdfb415fe5edc3d22f Nov 04 01:40:11.204256 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:11.204232 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:40:11.815487 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:11.810805 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5" event={"ID":"3934445a-0626-428a-a390-a49783fc7e73","Type":"ContainerStarted","Data":"3159fe5bb43bb54f9936684344ac5e628f764e6a966682fcb692c8ebc2ba4be8"} Nov 04 01:40:11.815487 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:11.810872 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5" event={"ID":"3934445a-0626-428a-a390-a49783fc7e73","Type":"ContainerStarted","Data":"2bee56fcbad4222f95ef940a393e36891e30a7305e3d50cdfb415fe5edc3d22f"} Nov 04 01:40:11.923151 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:11.923094 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5" podStartSLOduration=1.923078834 podStartE2EDuration="1.923078834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:40:11.921958035 +0000 UTC m=+1761.953691424" watchObservedRunningTime="2026-11-04 01:40:11.923078834 +0000 UTC m=+1761.954812223" Nov 04 01:40:12.826637 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:12.826601 5995 generic.go:356] "Generic (PLEG): container finished" podID="3934445a-0626-428a-a390-a49783fc7e73" containerID="3159fe5bb43bb54f9936684344ac5e628f764e6a966682fcb692c8ebc2ba4be8" exitCode=0 Nov 04 01:40:12.827675 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:12.827470 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5" event={"ID":"3934445a-0626-428a-a390-a49783fc7e73","Type":"ContainerDied","Data":"3159fe5bb43bb54f9936684344ac5e628f764e6a966682fcb692c8ebc2ba4be8"} Nov 04 01:40:14.078681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:14.078616 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" podUID="7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee" containerName="main" containerID="cri-o://c5e52b7aa75279cfd5443b22fdb3f23e42e27e54736d3bbf910cc1d90089f7d1" gracePeriod=45 Nov 04 01:40:14.842493 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:14.838202 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld"] Nov 04 01:40:14.926506 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:14.919844 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" podUID="7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": dial tcp 10.132.0.58:8000: connect: connection refused" Nov 04 01:40:15.079931 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:15.079898 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5" Nov 04 01:40:15.085436 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:15.084699 5995 generic.go:356] "Generic (PLEG): container finished" podID="7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee" containerID="c5e52b7aa75279cfd5443b22fdb3f23e42e27e54736d3bbf910cc1d90089f7d1" exitCode=0 Nov 04 01:40:15.085436 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:15.084874 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" event={"ID":"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee","Type":"ContainerDied","Data":"c5e52b7aa75279cfd5443b22fdb3f23e42e27e54736d3bbf910cc1d90089f7d1"} Nov 04 01:40:15.113310 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:15.112883 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5" event={"ID":"3934445a-0626-428a-a390-a49783fc7e73","Type":"ContainerDied","Data":"2bee56fcbad4222f95ef940a393e36891e30a7305e3d50cdfb415fe5edc3d22f"} Nov 04 01:40:15.113310 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:15.112948 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bee56fcbad4222f95ef940a393e36891e30a7305e3d50cdfb415fe5edc3d22f" Nov 04 01:40:15.113310 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:15.113024 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5" Nov 04 01:40:15.217172 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:15.217135 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3934445a-0626-428a-a390-a49783fc7e73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3934445a-0626-428a-a390-a49783fc7e73-maas-api-xks-ca\") pod \"3934445a-0626-428a-a390-a49783fc7e73\" (UID: \"3934445a-0626-428a-a390-a49783fc7e73\") " Nov 04 01:40:15.217559 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:15.217538 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3934445a-0626-428a-a390-a49783fc7e73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3934445a-0626-428a-a390-a49783fc7e73-maas-api-openshift-ca\") pod \"3934445a-0626-428a-a390-a49783fc7e73\" (UID: \"3934445a-0626-428a-a390-a49783fc7e73\") " Nov 04 01:40:15.217756 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:15.217736 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3934445a-0626-428a-a390-a49783fc7e73-kube-api-access-nvnnd\" (UniqueName: \"kubernetes.io/projected/3934445a-0626-428a-a390-a49783fc7e73-kube-api-access-nvnnd\") pod \"3934445a-0626-428a-a390-a49783fc7e73\" (UID: \"3934445a-0626-428a-a390-a49783fc7e73\") " Nov 04 01:40:15.234256 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:15.234199 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3934445a-0626-428a-a390-a49783fc7e73-maas-api-openshift-ca" pod "3934445a-0626-428a-a390-a49783fc7e73" (UID: "3934445a-0626-428a-a390-a49783fc7e73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:40:15.257490 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:15.247973 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3934445a-0626-428a-a390-a49783fc7e73-maas-api-xks-ca" pod "3934445a-0626-428a-a390-a49783fc7e73" (UID: "3934445a-0626-428a-a390-a49783fc7e73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:15.336174 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:15.335395 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3934445a-0626-428a-a390-a49783fc7e73-maas-api-openshift-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:15.336174 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:15.335455 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3934445a-0626-428a-a390-a49783fc7e73-maas-api-xks-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:15.354438 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:15.350050 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3934445a-0626-428a-a390-a49783fc7e73-kube-api-access-nvnnd" pod "3934445a-0626-428a-a390-a49783fc7e73" (UID: "3934445a-0626-428a-a390-a49783fc7e73"). InnerVolumeSpecName "kube-api-access-nvnnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:15.661497 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:15.650305 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvnnd\" (UniqueName: \"kubernetes.io/projected/3934445a-0626-428a-a390-a49783fc7e73-kube-api-access-nvnnd\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:15.864299 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:15.864245 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:40:16.039547 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.039516 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:40:16.098613 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.097804 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-home\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-home\") pod \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " Nov 04 01:40:16.098613 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.097877 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-dshm\") pod \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " Nov 04 01:40:16.098613 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.097965 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-tls-certs\") pod \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " Nov 04 01:40:16.098613 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.098030 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-kube-api-access-kgxbj\" (UniqueName: \"kubernetes.io/projected/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-kube-api-access-kgxbj\") pod \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " Nov 04 01:40:16.098613 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.098057 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-model-cache\") pod \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " Nov 04 01:40:16.098613 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.098088 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-tmp-dir\") pod \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\" (UID: \"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee\") " Nov 04 01:40:16.099935 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.099910 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-tmp-dir" pod "7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee" (UID: "7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:40:16.100279 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.100262 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-home" pod "7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee" (UID: "7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:40:16.101466 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.101433 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-model-cache" pod "7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee" (UID: "7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:40:16.139066 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.139040 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" Nov 04 01:40:16.140233 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.140205 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" event={"ID":"7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee","Type":"ContainerDied","Data":"c2d35ab2cb291d4b61eec050135c8680293cc18feebe1e115d05f7ef26709453"} Nov 04 01:40:16.140604 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.140589 5995 scope.go:122] "RemoveContainer" containerID="c5e52b7aa75279cfd5443b22fdb3f23e42e27e54736d3bbf910cc1d90089f7d1" Nov 04 01:40:16.175759 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.175719 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-kube-api-access-kgxbj" pod "7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee" (UID: "7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee"). InnerVolumeSpecName "kube-api-access-kgxbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:16.176162 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.176124 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-dshm" pod "7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee" (UID: "7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:40:16.181907 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.181849 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-tls-certs" pod "7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee" (UID: "7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:16.202162 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.201244 5995 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-home\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:16.202162 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.201277 5995 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-dshm\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:16.202162 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.201291 5995 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-tls-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:16.202162 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.201307 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgxbj\" (UniqueName: \"kubernetes.io/projected/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-kube-api-access-kgxbj\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:16.202162 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.201321 5995 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-model-cache\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:16.202162 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.201334 5995 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee-tmp-dir\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:16.205187 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.205028 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" podUID="0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b" containerName="main" containerID="cri-o://15502e66772ce886ef203af0b28ababf8f2156985e9d3027f24ff3a727f46b54" gracePeriod=45 Nov 04 01:40:16.211572 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:16.209067 5995 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3934445a_0626_428a_a390_a49783fc7e73.slice\": RecentStats: unable to find data in memory cache]" Nov 04 01:40:16.328439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.326672 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" podUID="0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": dial tcp 10.132.0.56:8000: connect: connection refused" Nov 04 01:40:16.578130 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.578095 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g"] Nov 04 01:40:16.581643 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.581613 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g"] Nov 04 01:40:16.612092 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.612071 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:40:16.705562 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.705527 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-model-cache\") pod \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " Nov 04 01:40:16.705754 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.705606 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-home\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-home\") pod \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " Nov 04 01:40:16.705754 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.705646 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-kube-api-access-8hsw5\" (UniqueName: \"kubernetes.io/projected/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-kube-api-access-8hsw5\") pod \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " Nov 04 01:40:16.705754 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.705680 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-tmp-dir\") pod \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " Nov 04 01:40:16.705925 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.705775 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-dshm\") pod \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " Nov 04 01:40:16.705925 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.705826 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-tls-certs\" (UniqueName: \"kubernetes.io/secret/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-tls-certs\") pod \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\" (UID: \"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b\") " Nov 04 01:40:16.705925 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.705868 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-model-cache" pod "0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b" (UID: "0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:40:16.705925 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.705902 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-home" pod "0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b" (UID: "0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:40:16.706099 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.706007 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-tmp-dir" pod "0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b" (UID: "0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:40:16.706182 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.706167 5995 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-tmp-dir\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:16.706280 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.706184 5995 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-model-cache\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:16.706280 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.706194 5995 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-home\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:16.707978 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.707948 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-dshm" pod "0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b" (UID: "0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:40:16.708085 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.708000 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-kube-api-access-8hsw5" pod "0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b" (UID: "0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b"). InnerVolumeSpecName "kube-api-access-8hsw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:16.708085 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.708051 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-tls-certs" pod "0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b" (UID: "0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:16.807356 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.807267 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hsw5\" (UniqueName: \"kubernetes.io/projected/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-kube-api-access-8hsw5\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:16.807356 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.807299 5995 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-dshm\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:16.807356 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.807313 5995 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b-tls-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:16.910014 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:16.909975 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b2eadb/premium-w4-b2eadb-kserve-847675f59d-2nc5g" podUID="7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:40:17.147206 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:17.147113 5995 generic.go:356] "Generic (PLEG): container finished" podID="0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b" containerID="15502e66772ce886ef203af0b28ababf8f2156985e9d3027f24ff3a727f46b54" exitCode=0 Nov 04 01:40:17.147206 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:17.147159 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" Nov 04 01:40:17.147800 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:17.147216 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" event={"ID":"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b","Type":"ContainerDied","Data":"15502e66772ce886ef203af0b28ababf8f2156985e9d3027f24ff3a727f46b54"} Nov 04 01:40:17.147800 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:17.147263 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v" event={"ID":"0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b","Type":"ContainerDied","Data":"bebe1de71b03c5f8888ab93997d1c3ed75ac90e3fb56a4df2157a3b637240eb2"} Nov 04 01:40:17.147800 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:17.147289 5995 scope.go:122] "RemoveContainer" containerID="15502e66772ce886ef203af0b28ababf8f2156985e9d3027f24ff3a727f46b54" Nov 04 01:40:17.157495 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:17.157476 5995 scope.go:122] "RemoveContainer" containerID="15502e66772ce886ef203af0b28ababf8f2156985e9d3027f24ff3a727f46b54" Nov 04 01:40:17.157757 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:17.157740 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15502e66772ce886ef203af0b28ababf8f2156985e9d3027f24ff3a727f46b54\": container with ID starting with 15502e66772ce886ef203af0b28ababf8f2156985e9d3027f24ff3a727f46b54 not found: ID does not exist" containerID="15502e66772ce886ef203af0b28ababf8f2156985e9d3027f24ff3a727f46b54" Nov 04 01:40:17.157823 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:17.157765 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15502e66772ce886ef203af0b28ababf8f2156985e9d3027f24ff3a727f46b54"} err="failed to get container status \"15502e66772ce886ef203af0b28ababf8f2156985e9d3027f24ff3a727f46b54\": rpc error: code = NotFound desc = could not find container \"15502e66772ce886ef203af0b28ababf8f2156985e9d3027f24ff3a727f46b54\": container with ID starting with 15502e66772ce886ef203af0b28ababf8f2156985e9d3027f24ff3a727f46b54 not found: ID does not exist" Nov 04 01:40:17.305577 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:17.305539 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v"] Nov 04 01:40:17.311904 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:17.311872 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-00cfe6/premium-w2-00cfe6-kserve-6888bb4c54-b4q4v"] Nov 04 01:40:18.129109 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:18.129076 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b" path="/var/lib/kubelet/pods/0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b/volumes" Nov 04 01:40:18.129532 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:18.129517 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee" path="/var/lib/kubelet/pods/7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee/volumes" Nov 04 01:40:19.641116 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:19.641079 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:40:19.839409 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:19.839344 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld" podUID="0ed35833-200d-4911-921f-910721ced9b2" containerName="payload-processing" containerID="cri-o://988e0b9d4773b7166382723d677494cff296a6f768cfef69df62ff3e9808f898" gracePeriod=25 Nov 04 01:40:20.090228 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:20.090202 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld" Nov 04 01:40:20.168398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:20.168365 5995 generic.go:356] "Generic (PLEG): container finished" podID="0ed35833-200d-4911-921f-910721ced9b2" containerID="988e0b9d4773b7166382723d677494cff296a6f768cfef69df62ff3e9808f898" exitCode=0 Nov 04 01:40:20.168582 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:20.168407 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld" event={"ID":"0ed35833-200d-4911-921f-910721ced9b2","Type":"ContainerDied","Data":"988e0b9d4773b7166382723d677494cff296a6f768cfef69df62ff3e9808f898"} Nov 04 01:40:20.168582 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:20.168458 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld" event={"ID":"0ed35833-200d-4911-921f-910721ced9b2","Type":"ContainerDied","Data":"6a8e2c25370f4b54eda3ef2336b7ac484900426466a6502db93b0df868501ada"} Nov 04 01:40:20.168582 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:20.168478 5995 scope.go:122] "RemoveContainer" containerID="988e0b9d4773b7166382723d677494cff296a6f768cfef69df62ff3e9808f898" Nov 04 01:40:20.168582 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:20.168409 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld" Nov 04 01:40:20.177770 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:20.177738 5995 scope.go:122] "RemoveContainer" containerID="988e0b9d4773b7166382723d677494cff296a6f768cfef69df62ff3e9808f898" Nov 04 01:40:20.178027 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:20.178010 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"988e0b9d4773b7166382723d677494cff296a6f768cfef69df62ff3e9808f898\": container with ID starting with 988e0b9d4773b7166382723d677494cff296a6f768cfef69df62ff3e9808f898 not found: ID does not exist" containerID="988e0b9d4773b7166382723d677494cff296a6f768cfef69df62ff3e9808f898" Nov 04 01:40:20.178079 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:20.178037 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988e0b9d4773b7166382723d677494cff296a6f768cfef69df62ff3e9808f898"} err="failed to get container status \"988e0b9d4773b7166382723d677494cff296a6f768cfef69df62ff3e9808f898\": rpc error: code = NotFound desc = could not find container \"988e0b9d4773b7166382723d677494cff296a6f768cfef69df62ff3e9808f898\": container with ID starting with 988e0b9d4773b7166382723d677494cff296a6f768cfef69df62ff3e9808f898 not found: ID does not exist" Nov 04 01:40:20.244896 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:20.244806 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed35833-200d-4911-921f-910721ced9b2-kube-api-access-mvvxk\" (UniqueName: \"kubernetes.io/projected/0ed35833-200d-4911-921f-910721ced9b2-kube-api-access-mvvxk\") pod \"0ed35833-200d-4911-921f-910721ced9b2\" (UID: \"0ed35833-200d-4911-921f-910721ced9b2\") " Nov 04 01:40:20.245055 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:20.244991 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed35833-200d-4911-921f-910721ced9b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed35833-200d-4911-921f-910721ced9b2-plugins-config-volume\") pod \"0ed35833-200d-4911-921f-910721ced9b2\" (UID: \"0ed35833-200d-4911-921f-910721ced9b2\") " Nov 04 01:40:20.245350 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:20.245326 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed35833-200d-4911-921f-910721ced9b2-plugins-config-volume" pod "0ed35833-200d-4911-921f-910721ced9b2" (UID: "0ed35833-200d-4911-921f-910721ced9b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:40:20.246889 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:20.246862 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed35833-200d-4911-921f-910721ced9b2-kube-api-access-mvvxk" pod "0ed35833-200d-4911-921f-910721ced9b2" (UID: "0ed35833-200d-4911-921f-910721ced9b2"). InnerVolumeSpecName "kube-api-access-mvvxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:20.346359 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:20.346317 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvvxk\" (UniqueName: \"kubernetes.io/projected/0ed35833-200d-4911-921f-910721ced9b2-kube-api-access-mvvxk\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:20.346359 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:20.346354 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed35833-200d-4911-921f-910721ced9b2-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:20.584125 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:20.584012 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld"] Nov 04 01:40:20.630708 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:20.630672 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b2eadb-7d764cd74-9j4ld"] Nov 04 01:40:22.128848 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:22.128812 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed35833-200d-4911-921f-910721ced9b2" path="/var/lib/kubelet/pods/0ed35833-200d-4911-921f-910721ced9b2/volumes" Nov 04 01:40:23.745713 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.745667 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2"] Nov 04 01:40:23.746479 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.746456 5995 cpu_manager.go:414] "containerMap: removing container" podUID="3934445a-0626-428a-a390-a49783fc7e73" containerName="revoke-keys" Nov 04 01:40:23.746602 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.746479 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3934445a-0626-428a-a390-a49783fc7e73" containerName="revoke-keys" Nov 04 01:40:23.746602 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.746511 5995 cpu_manager.go:414] "containerMap: removing container" podUID="0ed35833-200d-4911-921f-910721ced9b2" containerName="payload-processing" Nov 04 01:40:23.746602 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.746521 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed35833-200d-4911-921f-910721ced9b2" containerName="payload-processing" Nov 04 01:40:23.746602 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.746540 5995 cpu_manager.go:414] "containerMap: removing container" podUID="7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee" containerName="main" Nov 04 01:40:23.746602 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.746546 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee" containerName="main" Nov 04 01:40:23.746602 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.746577 5995 cpu_manager.go:414] "containerMap: removing container" podUID="0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b" containerName="main" Nov 04 01:40:23.746602 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.746585 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b" containerName="main" Nov 04 01:40:23.746937 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.746708 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="3934445a-0626-428a-a390-a49783fc7e73" containerName="revoke-keys" Nov 04 01:40:23.746937 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.746726 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed35833-200d-4911-921f-910721ced9b2" containerName="payload-processing" Nov 04 01:40:23.746937 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.746738 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f805d52-e8c2-4aa4-ab1c-3b52679eb1ee" containerName="main" Nov 04 01:40:23.746937 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.746752 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="0edbd841-3fd2-4f5c-ad82-e28ab6dabe3b" containerName="main" Nov 04 01:40:23.751611 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.751585 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2" Nov 04 01:40:23.782207 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.782175 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm7rm\" (UniqueName: \"kubernetes.io/projected/94b6da95-c060-4c58-8915-363619306374-kube-api-access-fm7rm\") pod \"maas-api-revoke-sub-dfa368ae-spfm2\" (UID: \"94b6da95-c060-4c58-8915-363619306374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2" Nov 04 01:40:23.782410 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.782213 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b6da95-c060-4c58-8915-363619306374-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfa368ae-spfm2\" (UID: \"94b6da95-c060-4c58-8915-363619306374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2" Nov 04 01:40:23.782505 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.782436 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b6da95-c060-4c58-8915-363619306374-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfa368ae-spfm2\" (UID: \"94b6da95-c060-4c58-8915-363619306374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2" Nov 04 01:40:23.805628 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.805581 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2"] Nov 04 01:40:23.846882 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.846849 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:40:23.846882 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.846858 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:40:23.846882 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.846857 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:40:23.847116 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.846916 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w799x\"" Nov 04 01:40:23.883386 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.883349 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b6da95-c060-4c58-8915-363619306374-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfa368ae-spfm2\" (UID: \"94b6da95-c060-4c58-8915-363619306374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2" Nov 04 01:40:23.883569 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.883493 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm7rm\" (UniqueName: \"kubernetes.io/projected/94b6da95-c060-4c58-8915-363619306374-kube-api-access-fm7rm\") pod \"maas-api-revoke-sub-dfa368ae-spfm2\" (UID: \"94b6da95-c060-4c58-8915-363619306374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2" Nov 04 01:40:23.883569 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.883523 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b6da95-c060-4c58-8915-363619306374-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfa368ae-spfm2\" (UID: \"94b6da95-c060-4c58-8915-363619306374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2" Nov 04 01:40:23.884197 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.884175 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b6da95-c060-4c58-8915-363619306374-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfa368ae-spfm2\" (UID: \"94b6da95-c060-4c58-8915-363619306374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2" Nov 04 01:40:23.885716 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.885689 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b6da95-c060-4c58-8915-363619306374-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfa368ae-spfm2\" (UID: \"94b6da95-c060-4c58-8915-363619306374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2" Nov 04 01:40:23.940908 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:23.940877 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm7rm\" (UniqueName: \"kubernetes.io/projected/94b6da95-c060-4c58-8915-363619306374-kube-api-access-fm7rm\") pod \"maas-api-revoke-sub-dfa368ae-spfm2\" (UID: \"94b6da95-c060-4c58-8915-363619306374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2" Nov 04 01:40:24.064498 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:24.064408 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2" Nov 04 01:40:24.462354 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:24.462317 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2"] Nov 04 01:40:24.477350 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:40:24.477299 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b6da95_c060_4c58_8915_363619306374.slice/crio-76725fcb151f14f7dc243c46aea21738e7ee58a7638d43fa99ec02029182fdfd WatchSource:0}: Error finding container 76725fcb151f14f7dc243c46aea21738e7ee58a7638d43fa99ec02029182fdfd: Status 404 returned error can't find the container with id 76725fcb151f14f7dc243c46aea21738e7ee58a7638d43fa99ec02029182fdfd Nov 04 01:40:25.264092 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:25.264050 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2" event={"ID":"94b6da95-c060-4c58-8915-363619306374","Type":"ContainerStarted","Data":"77f040ceeff0918819b0d31067b7230a5afb70c7c4c3e033826dab702e9db974"} Nov 04 01:40:25.264481 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:25.264116 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2" event={"ID":"94b6da95-c060-4c58-8915-363619306374","Type":"ContainerStarted","Data":"76725fcb151f14f7dc243c46aea21738e7ee58a7638d43fa99ec02029182fdfd"} Nov 04 01:40:25.373265 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:25.373212 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2" podStartSLOduration=2.373197357 podStartE2EDuration="2.373197357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:40:25.371996002 +0000 UTC m=+1775.403729386" watchObservedRunningTime="2026-11-04 01:40:25.373197357 +0000 UTC m=+1775.404930746" Nov 04 01:40:25.654245 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:25.654162 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:40:26.271658 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:26.271625 5995 generic.go:356] "Generic (PLEG): container finished" podID="94b6da95-c060-4c58-8915-363619306374" containerID="77f040ceeff0918819b0d31067b7230a5afb70c7c4c3e033826dab702e9db974" exitCode=0 Nov 04 01:40:26.272063 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:26.271709 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2" event={"ID":"94b6da95-c060-4c58-8915-363619306374","Type":"ContainerDied","Data":"77f040ceeff0918819b0d31067b7230a5afb70c7c4c3e033826dab702e9db974"} Nov 04 01:40:27.431944 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:27.431914 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2" Nov 04 01:40:27.541833 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:27.541563 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b6da95-c060-4c58-8915-363619306374-kube-api-access-fm7rm\" (UniqueName: \"kubernetes.io/projected/94b6da95-c060-4c58-8915-363619306374-kube-api-access-fm7rm\") pod \"94b6da95-c060-4c58-8915-363619306374\" (UID: \"94b6da95-c060-4c58-8915-363619306374\") " Nov 04 01:40:27.542107 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:27.542085 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94b6da95-c060-4c58-8915-363619306374-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b6da95-c060-4c58-8915-363619306374-maas-api-openshift-ca\") pod \"94b6da95-c060-4c58-8915-363619306374\" (UID: \"94b6da95-c060-4c58-8915-363619306374\") " Nov 04 01:40:27.542209 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:27.542146 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94b6da95-c060-4c58-8915-363619306374-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b6da95-c060-4c58-8915-363619306374-maas-api-xks-ca\") pod \"94b6da95-c060-4c58-8915-363619306374\" (UID: \"94b6da95-c060-4c58-8915-363619306374\") " Nov 04 01:40:27.542663 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:27.542635 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b6da95-c060-4c58-8915-363619306374-maas-api-openshift-ca" pod "94b6da95-c060-4c58-8915-363619306374" (UID: "94b6da95-c060-4c58-8915-363619306374"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:40:27.544695 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:27.544670 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b6da95-c060-4c58-8915-363619306374-kube-api-access-fm7rm" pod "94b6da95-c060-4c58-8915-363619306374" (UID: "94b6da95-c060-4c58-8915-363619306374"). InnerVolumeSpecName "kube-api-access-fm7rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:27.545056 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:27.545021 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b6da95-c060-4c58-8915-363619306374-maas-api-xks-ca" pod "94b6da95-c060-4c58-8915-363619306374" (UID: "94b6da95-c060-4c58-8915-363619306374"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:27.643724 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:27.643685 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b6da95-c060-4c58-8915-363619306374-maas-api-openshift-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:27.643724 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:27.643722 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b6da95-c060-4c58-8915-363619306374-maas-api-xks-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:27.643724 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:27.643733 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm7rm\" (UniqueName: \"kubernetes.io/projected/94b6da95-c060-4c58-8915-363619306374-kube-api-access-fm7rm\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:28.284748 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:28.284717 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2" Nov 04 01:40:28.284916 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:28.284756 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2" event={"ID":"94b6da95-c060-4c58-8915-363619306374","Type":"ContainerDied","Data":"76725fcb151f14f7dc243c46aea21738e7ee58a7638d43fa99ec02029182fdfd"} Nov 04 01:40:28.284916 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:28.284800 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76725fcb151f14f7dc243c46aea21738e7ee58a7638d43fa99ec02029182fdfd" Nov 04 01:40:33.055066 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:33.055033 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94"] Nov 04 01:40:33.101663 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:33.101630 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33933a5-c6x94"] Nov 04 01:40:34.129377 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:34.129342 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7feff85-eaf9-4643-8d8b-35ce61938408" path="/var/lib/kubelet/pods/d7feff85-eaf9-4643-8d8b-35ce61938408/volumes" Nov 04 01:40:35.654373 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:35.654339 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:40:45.654431 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:45.654380 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:40:48.048406 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:48.048372 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd"] Nov 04 01:40:48.094717 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:48.094680 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fa1c0a-btnvd"] Nov 04 01:40:48.129762 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:48.129729 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaafc194-ecb8-46a7-904e-f03823474419" path="/var/lib/kubelet/pods/eaafc194-ecb8-46a7-904e-f03823474419/volumes" Nov 04 01:40:49.642654 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:49.642615 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:40:50.048333 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:50.048299 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g"] Nov 04 01:40:50.094554 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:50.094524 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a80ef81-qwx4g"] Nov 04 01:40:50.131579 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:50.131545 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da967298-5193-4eb8-bbdc-6eaf8ecf5be3" path="/var/lib/kubelet/pods/da967298-5193-4eb8-bbdc-6eaf8ecf5be3/volumes" Nov 04 01:40:50.134458 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:50.132815 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ca76f974b55b: secret "agc-cred-ca76f974b55b" not found Nov 04 01:40:50.134458 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:50.132837 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7789777d6-l8vds: secret "agc-cred-ca76f974b55b" not found Nov 04 01:40:50.134458 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:50.133452 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials podName:95384e76-af24-412c-b18e-2177d638c7d4 nodeName:}" failed. No retries permitted until 2026-11-04 01:40:50.633430748 +0000 UTC m=+1800.665164117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials") pod "payload-processing-external-model-7789777d6-l8vds" (UID: "95384e76-af24-412c-b18e-2177d638c7d4") : secret "agc-cred-ca76f974b55b" not found Nov 04 01:40:50.349460 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:50.349363 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f_5600ddae-56c7-4c93-afff-2900207b0819/extract/0.log" Nov 04 01:40:50.351994 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:50.351971 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f_5600ddae-56c7-4c93-afff-2900207b0819/extract/0.log" Nov 04 01:40:50.359731 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:50.359703 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:40:50.362137 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:50.362115 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:40:50.637994 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:50.637913 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ca76f974b55b: secret "agc-cred-ca76f974b55b" not found Nov 04 01:40:50.637994 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:50.637940 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7789777d6-l8vds: secret "agc-cred-ca76f974b55b" not found Nov 04 01:40:50.638171 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:50.637998 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials podName:95384e76-af24-412c-b18e-2177d638c7d4 nodeName:}" failed. No retries permitted until 2026-11-04 01:40:51.637984456 +0000 UTC m=+1801.669717825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials") pod "payload-processing-external-model-7789777d6-l8vds" (UID: "95384e76-af24-412c-b18e-2177d638c7d4") : secret "agc-cred-ca76f974b55b" not found Nov 04 01:40:51.647951 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:51.647921 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ca76f974b55b: secret "agc-cred-ca76f974b55b" not found Nov 04 01:40:51.647951 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:51.647944 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7789777d6-l8vds: secret "agc-cred-ca76f974b55b" not found Nov 04 01:40:51.648362 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:51.647994 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials podName:95384e76-af24-412c-b18e-2177d638c7d4 nodeName:}" failed. No retries permitted until 2026-11-04 01:40:53.647980552 +0000 UTC m=+1803.679713920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials") pod "payload-processing-external-model-7789777d6-l8vds" (UID: "95384e76-af24-412c-b18e-2177d638c7d4") : secret "agc-cred-ca76f974b55b" not found Nov 04 01:40:52.479143 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:52.479111 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:40:53.687482 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:53.687446 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ca76f974b55b: secret "agc-cred-ca76f974b55b" not found Nov 04 01:40:53.687482 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:53.687479 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7789777d6-l8vds: secret "agc-cred-ca76f974b55b" not found Nov 04 01:40:53.688051 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:53.687549 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials podName:95384e76-af24-412c-b18e-2177d638c7d4 nodeName:}" failed. No retries permitted until 2026-11-04 01:40:57.68753017 +0000 UTC m=+1807.719263556 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials") pod "payload-processing-external-model-7789777d6-l8vds" (UID: "95384e76-af24-412c-b18e-2177d638c7d4") : secret "agc-cred-ca76f974b55b" not found Nov 04 01:40:55.522439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:55.521724 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-449zx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:40:55.522439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:55.521835 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" podUID="d392036c-7164-4328-9d05-a52e2e9eec82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:40:55.522439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:55.522205 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:40:55.522439 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:55.522267 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" podUID="370be073-d29b-47c0-8091-bee82e0b9dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:40:55.655877 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:55.655838 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:40:56.396308 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.396272 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6"] Nov 04 01:40:56.396840 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.396822 5995 cpu_manager.go:414] "containerMap: removing container" podUID="94b6da95-c060-4c58-8915-363619306374" containerName="revoke-keys" Nov 04 01:40:56.396938 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.396844 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b6da95-c060-4c58-8915-363619306374" containerName="revoke-keys" Nov 04 01:40:56.397076 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.397061 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b6da95-c060-4c58-8915-363619306374" containerName="revoke-keys" Nov 04 01:40:56.402472 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.402452 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.474890 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.474863 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-envoy\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.475012 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.474936 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-token\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.475056 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.475009 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-data\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.475056 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.475046 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-workload-socket\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.475165 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.475148 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-credential-socket\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.475215 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.475204 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnzr5\" (UniqueName: \"kubernetes.io/projected/91ff76ee-63ab-40a7-b7ef-8438908122ce-kube-api-access-cnzr5\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.475249 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.475230 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-podinfo\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.475284 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.475253 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ff76ee-63ab-40a7-b7ef-8438908122ce-istiod-ca-cert\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.475346 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.475334 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-workload-certs\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.504454 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.504405 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6"] Nov 04 01:40:56.576615 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.576580 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-credential-socket\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.577080 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.576653 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnzr5\" (UniqueName: \"kubernetes.io/projected/91ff76ee-63ab-40a7-b7ef-8438908122ce-kube-api-access-cnzr5\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.577080 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.576679 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-podinfo\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.577080 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.576699 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ff76ee-63ab-40a7-b7ef-8438908122ce-istiod-ca-cert\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.577080 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.576729 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-workload-certs\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.577080 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.576766 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-envoy\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.577080 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.576858 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-token\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.577080 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.576897 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-data\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.577080 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.576936 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-workload-socket\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.577080 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.577071 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-credential-socket\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.577585 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.577496 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ff76ee-63ab-40a7-b7ef-8438908122ce-istiod-ca-cert\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.577641 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.577582 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-workload-certs\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.577641 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.577630 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-data\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.577749 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.577723 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-workload-socket\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.579239 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.579216 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-envoy\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.579331 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.579315 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-podinfo\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.634450 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.634385 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-token\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.634605 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.634512 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnzr5\" (UniqueName: \"kubernetes.io/projected/91ff76ee-63ab-40a7-b7ef-8438908122ce-kube-api-access-cnzr5\") pod \"e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:56.715760 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:56.715731 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:57.143784 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:57.143696 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6"] Nov 04 01:40:57.148857 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:40:57.148830 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ff76ee_63ab_40a7_b7ef_8438908122ce.slice/crio-fbb08058ea5bb9ac4eb40782fe21a38c39c571806ee9481a8ff8a1bc4399f591 WatchSource:0}: Error finding container fbb08058ea5bb9ac4eb40782fe21a38c39c571806ee9481a8ff8a1bc4399f591: Status 404 returned error can't find the container with id fbb08058ea5bb9ac4eb40782fe21a38c39c571806ee9481a8ff8a1bc4399f591 Nov 04 01:40:57.151076 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:57.151044 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:40:57.151160 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:57.151113 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:40:57.151160 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:57.151142 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:40:57.550393 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:57.550355 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" event={"ID":"91ff76ee-63ab-40a7-b7ef-8438908122ce","Type":"ContainerStarted","Data":"95528de5524fe8ef1ed3eecdbed87b91c1c2c691fb8b2dcf41c4231172ad82de"} Nov 04 01:40:57.550622 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:57.550448 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" event={"ID":"91ff76ee-63ab-40a7-b7ef-8438908122ce","Type":"ContainerStarted","Data":"fbb08058ea5bb9ac4eb40782fe21a38c39c571806ee9481a8ff8a1bc4399f591"} Nov 04 01:40:57.666097 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:57.666046 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" podStartSLOduration=1.666031724 podStartE2EDuration="1.666031724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:40:57.664817306 +0000 UTC m=+1807.696550696" watchObservedRunningTime="2026-11-04 01:40:57.666031724 +0000 UTC m=+1807.697765114" Nov 04 01:40:57.703044 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:57.703010 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ca76f974b55b: secret "agc-cred-ca76f974b55b" not found Nov 04 01:40:57.703044 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:57.703033 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7789777d6-l8vds: secret "agc-cred-ca76f974b55b" not found Nov 04 01:40:57.703249 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:40:57.703083 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials podName:95384e76-af24-412c-b18e-2177d638c7d4 nodeName:}" failed. No retries permitted until 2026-11-04 01:41:05.703069215 +0000 UTC m=+1815.734802587 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials") pod "payload-processing-external-model-7789777d6-l8vds" (UID: "95384e76-af24-412c-b18e-2177d638c7d4") : secret "agc-cred-ca76f974b55b" not found Nov 04 01:40:57.716693 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:57.716663 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:58.721534 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:58.721456 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:59.563011 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:59.562984 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:40:59.563972 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:40:59.563954 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:41:05.653976 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:05.653943 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:41:05.794702 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:41:05.794670 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ca76f974b55b: secret "agc-cred-ca76f974b55b" not found Nov 04 01:41:05.794702 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:41:05.794696 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7789777d6-l8vds: secret "agc-cred-ca76f974b55b" not found Nov 04 01:41:05.794934 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:41:05.794762 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials podName:95384e76-af24-412c-b18e-2177d638c7d4 nodeName:}" failed. No retries permitted until 2026-11-04 01:41:21.794746239 +0000 UTC m=+1831.826479613 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials") pod "payload-processing-external-model-7789777d6-l8vds" (UID: "95384e76-af24-412c-b18e-2177d638c7d4") : secret "agc-cred-ca76f974b55b" not found Nov 04 01:41:07.046703 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:07.046667 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9"] Nov 04 01:41:07.093249 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:07.093216 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00177864-jg8m9"] Nov 04 01:41:07.716033 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:07.715995 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6"] Nov 04 01:41:07.716277 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:07.716236 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" podUID="91ff76ee-63ab-40a7-b7ef-8438908122ce" containerName="istio-proxy" containerID="cri-o://95528de5524fe8ef1ed3eecdbed87b91c1c2c691fb8b2dcf41c4231172ad82de" gracePeriod=30 Nov 04 01:41:07.737587 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:07.737554 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2"] Nov 04 01:41:07.737790 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:07.737757 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" podUID="9d541a44-f6eb-47ae-b9c5-244f33af1bf1" containerName="istio-proxy" containerID="cri-o://04a29f375fa515e212de6fa3ec827a8bcb2cc610bd51e08363bbfc976a2882cf" gracePeriod=30 Nov 04 01:41:08.129286 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:08.129200 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd33b9f-6727-4178-b340-6845fe234a1d" path="/var/lib/kubelet/pods/bdd33b9f-6727-4178-b340-6845fe234a1d/volumes" Nov 04 01:41:08.271956 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:08.271919 5995 patch_prober.go:29] interesting pod/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:41:08.272128 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:08.271986 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" podUID="9d541a44-f6eb-47ae-b9c5-244f33af1bf1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:41:12.991933 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:12.991860 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:41:13.036722 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.036700 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:41:13.085061 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085026 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-podinfo\") pod \"91ff76ee-63ab-40a7-b7ef-8438908122ce\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " Nov 04 01:41:13.085061 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085064 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-token\" (UniqueName: \"kubernetes.io/projected/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-token\") pod \"91ff76ee-63ab-40a7-b7ef-8438908122ce\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " Nov 04 01:41:13.085254 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085089 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-credential-socket\") pod \"91ff76ee-63ab-40a7-b7ef-8438908122ce\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " Nov 04 01:41:13.085254 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085205 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-credential-socket\") pod \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " Nov 04 01:41:13.085254 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085251 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-data\") pod \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " Nov 04 01:41:13.085395 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085288 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-workload-socket\") pod \"91ff76ee-63ab-40a7-b7ef-8438908122ce\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " Nov 04 01:41:13.085395 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085315 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ff76ee-63ab-40a7-b7ef-8438908122ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ff76ee-63ab-40a7-b7ef-8438908122ce-istiod-ca-cert\") pod \"91ff76ee-63ab-40a7-b7ef-8438908122ce\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " Nov 04 01:41:13.085395 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085344 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-credential-socket" pod "91ff76ee-63ab-40a7-b7ef-8438908122ce" (UID: "91ff76ee-63ab-40a7-b7ef-8438908122ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:41:13.085395 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085344 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-podinfo\") pod \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " Nov 04 01:41:13.085614 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085493 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-envoy\") pod \"91ff76ee-63ab-40a7-b7ef-8438908122ce\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " Nov 04 01:41:13.085614 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085532 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-data\") pod \"91ff76ee-63ab-40a7-b7ef-8438908122ce\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " Nov 04 01:41:13.085614 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085542 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-data" pod "9d541a44-f6eb-47ae-b9c5-244f33af1bf1" (UID: "9d541a44-f6eb-47ae-b9c5-244f33af1bf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:41:13.085614 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085587 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istiod-ca-cert\") pod \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " Nov 04 01:41:13.085810 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085619 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-envoy\") pod \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " Nov 04 01:41:13.085810 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085656 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-workload-certs\") pod \"91ff76ee-63ab-40a7-b7ef-8438908122ce\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " Nov 04 01:41:13.085810 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085691 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ff76ee-63ab-40a7-b7ef-8438908122ce-kube-api-access-cnzr5\" (UniqueName: \"kubernetes.io/projected/91ff76ee-63ab-40a7-b7ef-8438908122ce-kube-api-access-cnzr5\") pod \"91ff76ee-63ab-40a7-b7ef-8438908122ce\" (UID: \"91ff76ee-63ab-40a7-b7ef-8438908122ce\") " Nov 04 01:41:13.085810 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085725 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-token\" (UniqueName: \"kubernetes.io/projected/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-token\") pod \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " Nov 04 01:41:13.085810 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085757 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-kube-api-access-g8jlz\" (UniqueName: \"kubernetes.io/projected/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-kube-api-access-g8jlz\") pod \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " Nov 04 01:41:13.086062 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085821 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-workload-certs\") pod \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " Nov 04 01:41:13.086062 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085881 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-workload-socket\") pod \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\" (UID: \"9d541a44-f6eb-47ae-b9c5-244f33af1bf1\") " Nov 04 01:41:13.086062 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085616 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-workload-socket" pod "91ff76ee-63ab-40a7-b7ef-8438908122ce" (UID: "91ff76ee-63ab-40a7-b7ef-8438908122ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:41:13.086062 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085882 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-credential-socket" pod "9d541a44-f6eb-47ae-b9c5-244f33af1bf1" (UID: "9d541a44-f6eb-47ae-b9c5-244f33af1bf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:41:13.086062 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.085949 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istiod-ca-cert" pod "9d541a44-f6eb-47ae-b9c5-244f33af1bf1" (UID: "9d541a44-f6eb-47ae-b9c5-244f33af1bf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:41:13.086337 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.086088 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ff76ee-63ab-40a7-b7ef-8438908122ce-istiod-ca-cert" pod "91ff76ee-63ab-40a7-b7ef-8438908122ce" (UID: "91ff76ee-63ab-40a7-b7ef-8438908122ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:41:13.086337 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.086288 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-data" pod "91ff76ee-63ab-40a7-b7ef-8438908122ce" (UID: "91ff76ee-63ab-40a7-b7ef-8438908122ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:41:13.086549 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.086518 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-workload-certs" pod "9d541a44-f6eb-47ae-b9c5-244f33af1bf1" (UID: "9d541a44-f6eb-47ae-b9c5-244f33af1bf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:41:13.086629 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.086571 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-workload-certs" pod "91ff76ee-63ab-40a7-b7ef-8438908122ce" (UID: "91ff76ee-63ab-40a7-b7ef-8438908122ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:41:13.086684 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.086620 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-workload-socket" pod "9d541a44-f6eb-47ae-b9c5-244f33af1bf1" (UID: "9d541a44-f6eb-47ae-b9c5-244f33af1bf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:41:13.086766 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.086746 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:13.086822 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.086772 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:13.086822 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.086788 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:13.086822 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.086802 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:13.086822 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.086815 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ff76ee-63ab-40a7-b7ef-8438908122ce-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:13.087073 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.086827 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:13.087073 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.086840 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:13.087073 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.086854 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:13.087073 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.086869 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:13.087073 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.086885 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:13.088203 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.087855 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-token" pod "91ff76ee-63ab-40a7-b7ef-8438908122ce" (UID: "91ff76ee-63ab-40a7-b7ef-8438908122ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:41:13.088361 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.088251 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-podinfo" pod "9d541a44-f6eb-47ae-b9c5-244f33af1bf1" (UID: "9d541a44-f6eb-47ae-b9c5-244f33af1bf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:41:13.088361 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.088345 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-podinfo" pod "91ff76ee-63ab-40a7-b7ef-8438908122ce" (UID: "91ff76ee-63ab-40a7-b7ef-8438908122ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:41:13.088501 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.088470 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-envoy" pod "9d541a44-f6eb-47ae-b9c5-244f33af1bf1" (UID: "9d541a44-f6eb-47ae-b9c5-244f33af1bf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:41:13.088658 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.088642 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-envoy" pod "91ff76ee-63ab-40a7-b7ef-8438908122ce" (UID: "91ff76ee-63ab-40a7-b7ef-8438908122ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:41:13.088732 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.088715 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-kube-api-access-g8jlz" pod "9d541a44-f6eb-47ae-b9c5-244f33af1bf1" (UID: "9d541a44-f6eb-47ae-b9c5-244f33af1bf1"). InnerVolumeSpecName "kube-api-access-g8jlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:41:13.088789 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.088770 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ff76ee-63ab-40a7-b7ef-8438908122ce-kube-api-access-cnzr5" pod "91ff76ee-63ab-40a7-b7ef-8438908122ce" (UID: "91ff76ee-63ab-40a7-b7ef-8438908122ce"). InnerVolumeSpecName "kube-api-access-cnzr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:41:13.089473 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.089455 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-token" pod "9d541a44-f6eb-47ae-b9c5-244f33af1bf1" (UID: "9d541a44-f6eb-47ae-b9c5-244f33af1bf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:41:13.188047 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.188016 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnzr5\" (UniqueName: \"kubernetes.io/projected/91ff76ee-63ab-40a7-b7ef-8438908122ce-kube-api-access-cnzr5\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:13.188047 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.188046 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:13.188047 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.188055 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8jlz\" (UniqueName: \"kubernetes.io/projected/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-kube-api-access-g8jlz\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:13.188267 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.188065 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:13.188267 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.188073 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:13.188267 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.188081 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:13.188267 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.188089 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ff76ee-63ab-40a7-b7ef-8438908122ce-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:13.188267 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.188097 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d541a44-f6eb-47ae-b9c5-244f33af1bf1-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:13.649595 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.649562 5995 generic.go:356] "Generic (PLEG): container finished" podID="91ff76ee-63ab-40a7-b7ef-8438908122ce" containerID="95528de5524fe8ef1ed3eecdbed87b91c1c2c691fb8b2dcf41c4231172ad82de" exitCode=0 Nov 04 01:41:13.649776 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.649608 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" Nov 04 01:41:13.649776 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.649649 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" event={"ID":"91ff76ee-63ab-40a7-b7ef-8438908122ce","Type":"ContainerDied","Data":"95528de5524fe8ef1ed3eecdbed87b91c1c2c691fb8b2dcf41c4231172ad82de"} Nov 04 01:41:13.649776 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.649702 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6" event={"ID":"91ff76ee-63ab-40a7-b7ef-8438908122ce","Type":"ContainerDied","Data":"fbb08058ea5bb9ac4eb40782fe21a38c39c571806ee9481a8ff8a1bc4399f591"} Nov 04 01:41:13.649776 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.649728 5995 scope.go:122] "RemoveContainer" containerID="95528de5524fe8ef1ed3eecdbed87b91c1c2c691fb8b2dcf41c4231172ad82de" Nov 04 01:41:13.651480 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.651459 5995 generic.go:356] "Generic (PLEG): container finished" podID="9d541a44-f6eb-47ae-b9c5-244f33af1bf1" containerID="04a29f375fa515e212de6fa3ec827a8bcb2cc610bd51e08363bbfc976a2882cf" exitCode=0 Nov 04 01:41:13.651579 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.651519 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" Nov 04 01:41:13.651632 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.651562 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" event={"ID":"9d541a44-f6eb-47ae-b9c5-244f33af1bf1","Type":"ContainerDied","Data":"04a29f375fa515e212de6fa3ec827a8bcb2cc610bd51e08363bbfc976a2882cf"} Nov 04 01:41:13.651665 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.651634 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2" event={"ID":"9d541a44-f6eb-47ae-b9c5-244f33af1bf1","Type":"ContainerDied","Data":"b9c684c9623011f827fc5eb6d0112c6f4d5f622b10db0280f0a626c4a56539fb"} Nov 04 01:41:13.659475 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.659451 5995 scope.go:122] "RemoveContainer" containerID="95528de5524fe8ef1ed3eecdbed87b91c1c2c691fb8b2dcf41c4231172ad82de" Nov 04 01:41:13.659805 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:41:13.659773 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95528de5524fe8ef1ed3eecdbed87b91c1c2c691fb8b2dcf41c4231172ad82de\": container with ID starting with 95528de5524fe8ef1ed3eecdbed87b91c1c2c691fb8b2dcf41c4231172ad82de not found: ID does not exist" containerID="95528de5524fe8ef1ed3eecdbed87b91c1c2c691fb8b2dcf41c4231172ad82de" Nov 04 01:41:13.659901 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.659806 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95528de5524fe8ef1ed3eecdbed87b91c1c2c691fb8b2dcf41c4231172ad82de"} err="failed to get container status \"95528de5524fe8ef1ed3eecdbed87b91c1c2c691fb8b2dcf41c4231172ad82de\": rpc error: code = NotFound desc = could not find container \"95528de5524fe8ef1ed3eecdbed87b91c1c2c691fb8b2dcf41c4231172ad82de\": container with ID starting with 95528de5524fe8ef1ed3eecdbed87b91c1c2c691fb8b2dcf41c4231172ad82de not found: ID does not exist" Nov 04 01:41:13.659901 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.659829 5995 scope.go:122] "RemoveContainer" containerID="04a29f375fa515e212de6fa3ec827a8bcb2cc610bd51e08363bbfc976a2882cf" Nov 04 01:41:13.668864 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.668845 5995 scope.go:122] "RemoveContainer" containerID="f019f7d3e19ef088515cc7f289fa2e80ac38c06595c0d04207e2df5be79b32c9" Nov 04 01:41:13.677225 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.677207 5995 scope.go:122] "RemoveContainer" containerID="04a29f375fa515e212de6fa3ec827a8bcb2cc610bd51e08363bbfc976a2882cf" Nov 04 01:41:13.677502 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:41:13.677485 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04a29f375fa515e212de6fa3ec827a8bcb2cc610bd51e08363bbfc976a2882cf\": container with ID starting with 04a29f375fa515e212de6fa3ec827a8bcb2cc610bd51e08363bbfc976a2882cf not found: ID does not exist" containerID="04a29f375fa515e212de6fa3ec827a8bcb2cc610bd51e08363bbfc976a2882cf" Nov 04 01:41:13.677562 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.677509 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a29f375fa515e212de6fa3ec827a8bcb2cc610bd51e08363bbfc976a2882cf"} err="failed to get container status \"04a29f375fa515e212de6fa3ec827a8bcb2cc610bd51e08363bbfc976a2882cf\": rpc error: code = NotFound desc = could not find container \"04a29f375fa515e212de6fa3ec827a8bcb2cc610bd51e08363bbfc976a2882cf\": container with ID starting with 04a29f375fa515e212de6fa3ec827a8bcb2cc610bd51e08363bbfc976a2882cf not found: ID does not exist" Nov 04 01:41:13.677562 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.677525 5995 scope.go:122] "RemoveContainer" containerID="f019f7d3e19ef088515cc7f289fa2e80ac38c06595c0d04207e2df5be79b32c9" Nov 04 01:41:13.677756 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:41:13.677736 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f019f7d3e19ef088515cc7f289fa2e80ac38c06595c0d04207e2df5be79b32c9\": container with ID starting with f019f7d3e19ef088515cc7f289fa2e80ac38c06595c0d04207e2df5be79b32c9 not found: ID does not exist" containerID="f019f7d3e19ef088515cc7f289fa2e80ac38c06595c0d04207e2df5be79b32c9" Nov 04 01:41:13.677797 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.677766 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f019f7d3e19ef088515cc7f289fa2e80ac38c06595c0d04207e2df5be79b32c9"} err="failed to get container status \"f019f7d3e19ef088515cc7f289fa2e80ac38c06595c0d04207e2df5be79b32c9\": rpc error: code = NotFound desc = could not find container \"f019f7d3e19ef088515cc7f289fa2e80ac38c06595c0d04207e2df5be79b32c9\": container with ID starting with f019f7d3e19ef088515cc7f289fa2e80ac38c06595c0d04207e2df5be79b32c9 not found: ID does not exist" Nov 04 01:41:13.804778 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.804753 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6"] Nov 04 01:41:13.851197 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.851169 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-xl9n6"] Nov 04 01:41:13.911843 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.911769 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2"] Nov 04 01:41:13.915868 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:13.915841 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b2eadb-openshift-default-54d5bd64fc-2ftq2"] Nov 04 01:41:14.129526 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:14.129493 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ff76ee-63ab-40a7-b7ef-8438908122ce" path="/var/lib/kubelet/pods/91ff76ee-63ab-40a7-b7ef-8438908122ce/volumes" Nov 04 01:41:14.129955 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:14.129941 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d541a44-f6eb-47ae-b9c5-244f33af1bf1" path="/var/lib/kubelet/pods/9d541a44-f6eb-47ae-b9c5-244f33af1bf1/volumes" Nov 04 01:41:15.653690 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:15.653653 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:41:19.642596 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:41:19.642558 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:41:21.881975 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:41:21.881932 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ca76f974b55b: secret "agc-cred-ca76f974b55b" not found Nov 04 01:41:21.881975 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:41:21.881967 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7789777d6-l8vds: secret "agc-cred-ca76f974b55b" not found Nov 04 01:41:21.882450 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:41:21.882033 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials podName:95384e76-af24-412c-b18e-2177d638c7d4 nodeName:}" failed. No retries permitted until 2026-11-04 01:41:53.882017012 +0000 UTC m=+1863.913750389 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials") pod "payload-processing-external-model-7789777d6-l8vds" (UID: "95384e76-af24-412c-b18e-2177d638c7d4") : secret "agc-cred-ca76f974b55b" not found Nov 04 01:41:25.123330 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:25.123286 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:41:35.124020 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:35.123984 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:41:45.123474 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:45.123437 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:41:49.641728 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:41:49.641684 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:41:50.565845 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:50.565804 5995 scope.go:122] "RemoveContainer" containerID="a10e051d6eb2310000f4fbf625cda1774de124e0ca05f79c5418527a06d8e816" Nov 04 01:41:50.575577 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:50.575553 5995 scope.go:122] "RemoveContainer" containerID="48fee41e63f42d8e386efd7ef64a212338df50df126bacd14b547a9337b428e4" Nov 04 01:41:50.584319 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:50.584296 5995 scope.go:122] "RemoveContainer" containerID="66dbbc060ef9726a951a65841e76e9a9c4121e06ffa9eb5f062fdc84d00262e5" Nov 04 01:41:53.924659 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:41:53.924621 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ca76f974b55b: secret "agc-cred-ca76f974b55b" not found Nov 04 01:41:53.924659 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:41:53.924650 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7789777d6-l8vds: secret "agc-cred-ca76f974b55b" not found Nov 04 01:41:53.925180 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:41:53.924727 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials podName:95384e76-af24-412c-b18e-2177d638c7d4 nodeName:}" failed. No retries permitted until 2026-11-04 01:42:57.924707842 +0000 UTC m=+1927.956441210 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials") pod "payload-processing-external-model-7789777d6-l8vds" (UID: "95384e76-af24-412c-b18e-2177d638c7d4") : secret "agc-cred-ca76f974b55b" not found Nov 04 01:41:55.123288 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:55.123253 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:41:56.036410 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:56.036372 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj"] Nov 04 01:41:56.036621 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:56.036590 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" podUID="12797bc0-f8ab-4176-a807-1a3918fb8d0e" containerName="istio-proxy" containerID="cri-o://8704bc717555d5e2febf5c97e83fe0a870f880f4fb077f90a923f89532391593" gracePeriod=30 Nov 04 01:41:57.537938 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:57.537901 5995 patch_prober.go:29] interesting pod/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:41:57.538335 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:41:57.537961 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" podUID="12797bc0-f8ab-4176-a807-1a3918fb8d0e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:42:01.302655 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.302587 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:42:01.424636 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.424604 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-workload-socket\") pod \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " Nov 04 01:42:01.424839 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.424643 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-credential-socket\") pod \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " Nov 04 01:42:01.424839 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.424682 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istiod-ca-cert\") pod \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " Nov 04 01:42:01.424839 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.424726 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-workload-certs\") pod \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " Nov 04 01:42:01.424839 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.424747 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-token\" (UniqueName: \"kubernetes.io/projected/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-token\") pod \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " Nov 04 01:42:01.424839 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.424781 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-data\") pod \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " Nov 04 01:42:01.425105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.424853 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-envoy\") pod \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " Nov 04 01:42:01.425105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.424891 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12797bc0-f8ab-4176-a807-1a3918fb8d0e-kube-api-access-rdpt4\" (UniqueName: \"kubernetes.io/projected/12797bc0-f8ab-4176-a807-1a3918fb8d0e-kube-api-access-rdpt4\") pod \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " Nov 04 01:42:01.425105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.424916 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-workload-socket" pod "12797bc0-f8ab-4176-a807-1a3918fb8d0e" (UID: "12797bc0-f8ab-4176-a807-1a3918fb8d0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:42:01.425105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.424981 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-podinfo\") pod \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\" (UID: \"12797bc0-f8ab-4176-a807-1a3918fb8d0e\") " Nov 04 01:42:01.425105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.424992 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-credential-socket" pod "12797bc0-f8ab-4176-a807-1a3918fb8d0e" (UID: "12797bc0-f8ab-4176-a807-1a3918fb8d0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:42:01.425105 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.425086 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-workload-certs" pod "12797bc0-f8ab-4176-a807-1a3918fb8d0e" (UID: "12797bc0-f8ab-4176-a807-1a3918fb8d0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:42:01.425446 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.425301 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istiod-ca-cert" pod "12797bc0-f8ab-4176-a807-1a3918fb8d0e" (UID: "12797bc0-f8ab-4176-a807-1a3918fb8d0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:42:01.425446 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.425297 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-data" pod "12797bc0-f8ab-4176-a807-1a3918fb8d0e" (UID: "12797bc0-f8ab-4176-a807-1a3918fb8d0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:42:01.425446 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.425402 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:01.425446 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.425441 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:01.425666 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.425456 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:01.425666 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.425469 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:01.425666 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.425480 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:01.427135 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.427098 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-podinfo" pod "12797bc0-f8ab-4176-a807-1a3918fb8d0e" (UID: "12797bc0-f8ab-4176-a807-1a3918fb8d0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:42:01.427254 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.427197 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12797bc0-f8ab-4176-a807-1a3918fb8d0e-kube-api-access-rdpt4" pod "12797bc0-f8ab-4176-a807-1a3918fb8d0e" (UID: "12797bc0-f8ab-4176-a807-1a3918fb8d0e"). InnerVolumeSpecName "kube-api-access-rdpt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:42:01.427254 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.427225 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-token" pod "12797bc0-f8ab-4176-a807-1a3918fb8d0e" (UID: "12797bc0-f8ab-4176-a807-1a3918fb8d0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:42:01.427254 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.427222 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-envoy" pod "12797bc0-f8ab-4176-a807-1a3918fb8d0e" (UID: "12797bc0-f8ab-4176-a807-1a3918fb8d0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:42:01.526541 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.526503 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:01.526541 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.526535 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:01.526541 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.526546 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdpt4\" (UniqueName: \"kubernetes.io/projected/12797bc0-f8ab-4176-a807-1a3918fb8d0e-kube-api-access-rdpt4\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:01.526768 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.526557 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12797bc0-f8ab-4176-a807-1a3918fb8d0e-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:01.953755 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.953714 5995 generic.go:356] "Generic (PLEG): container finished" podID="12797bc0-f8ab-4176-a807-1a3918fb8d0e" containerID="8704bc717555d5e2febf5c97e83fe0a870f880f4fb077f90a923f89532391593" exitCode=0 Nov 04 01:42:01.953935 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.953769 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" Nov 04 01:42:01.953935 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.953790 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" event={"ID":"12797bc0-f8ab-4176-a807-1a3918fb8d0e","Type":"ContainerDied","Data":"8704bc717555d5e2febf5c97e83fe0a870f880f4fb077f90a923f89532391593"} Nov 04 01:42:01.953935 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.953843 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj" event={"ID":"12797bc0-f8ab-4176-a807-1a3918fb8d0e","Type":"ContainerDied","Data":"4c917ade57127bf6cffa24ee7416a48d850ecbdc7d3d8629b9bf4b77dc9e58f9"} Nov 04 01:42:01.953935 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.953865 5995 scope.go:122] "RemoveContainer" containerID="8704bc717555d5e2febf5c97e83fe0a870f880f4fb077f90a923f89532391593" Nov 04 01:42:01.966915 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.966891 5995 scope.go:122] "RemoveContainer" containerID="a5093bcbffa94a7ea26cae640fe5d8922d879ccc9c720d6cc0892aa2afa63591" Nov 04 01:42:01.975990 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.975969 5995 scope.go:122] "RemoveContainer" containerID="8704bc717555d5e2febf5c97e83fe0a870f880f4fb077f90a923f89532391593" Nov 04 01:42:01.976288 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:42:01.976266 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8704bc717555d5e2febf5c97e83fe0a870f880f4fb077f90a923f89532391593\": container with ID starting with 8704bc717555d5e2febf5c97e83fe0a870f880f4fb077f90a923f89532391593 not found: ID does not exist" containerID="8704bc717555d5e2febf5c97e83fe0a870f880f4fb077f90a923f89532391593" Nov 04 01:42:01.976370 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.976302 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8704bc717555d5e2febf5c97e83fe0a870f880f4fb077f90a923f89532391593"} err="failed to get container status \"8704bc717555d5e2febf5c97e83fe0a870f880f4fb077f90a923f89532391593\": rpc error: code = NotFound desc = could not find container \"8704bc717555d5e2febf5c97e83fe0a870f880f4fb077f90a923f89532391593\": container with ID starting with 8704bc717555d5e2febf5c97e83fe0a870f880f4fb077f90a923f89532391593 not found: ID does not exist" Nov 04 01:42:01.976370 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.976327 5995 scope.go:122] "RemoveContainer" containerID="a5093bcbffa94a7ea26cae640fe5d8922d879ccc9c720d6cc0892aa2afa63591" Nov 04 01:42:01.976607 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:42:01.976589 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5093bcbffa94a7ea26cae640fe5d8922d879ccc9c720d6cc0892aa2afa63591\": container with ID starting with a5093bcbffa94a7ea26cae640fe5d8922d879ccc9c720d6cc0892aa2afa63591 not found: ID does not exist" containerID="a5093bcbffa94a7ea26cae640fe5d8922d879ccc9c720d6cc0892aa2afa63591" Nov 04 01:42:01.976660 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:01.976616 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5093bcbffa94a7ea26cae640fe5d8922d879ccc9c720d6cc0892aa2afa63591"} err="failed to get container status \"a5093bcbffa94a7ea26cae640fe5d8922d879ccc9c720d6cc0892aa2afa63591\": rpc error: code = NotFound desc = could not find container \"a5093bcbffa94a7ea26cae640fe5d8922d879ccc9c720d6cc0892aa2afa63591\": container with ID starting with a5093bcbffa94a7ea26cae640fe5d8922d879ccc9c720d6cc0892aa2afa63591 not found: ID does not exist" Nov 04 01:42:02.156831 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:02.156794 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj"] Nov 04 01:42:02.203311 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:02.203279 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-mbxzj"] Nov 04 01:42:04.128112 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:04.128081 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12797bc0-f8ab-4176-a807-1a3918fb8d0e" path="/var/lib/kubelet/pods/12797bc0-f8ab-4176-a807-1a3918fb8d0e/volumes" Nov 04 01:42:05.123469 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:05.123433 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:42:15.123505 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:15.123466 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:42:19.640988 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:42:19.640945 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:42:23.046763 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:23.046722 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt"] Nov 04 01:42:23.093706 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:23.093670 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad94dfa-zw5kt"] Nov 04 01:42:24.128771 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:24.128733 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77044b62-3256-44be-b8d7-245046a12ff1" path="/var/lib/kubelet/pods/77044b62-3256-44be-b8d7-245046a12ff1/volumes" Nov 04 01:42:25.124260 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:25.124216 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:42:35.047190 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:35.047105 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv"] Nov 04 01:42:35.093624 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:35.093588 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71e1959d-8f8hv"] Nov 04 01:42:35.123411 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:35.123381 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:42:36.128839 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:36.128806 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46569cf9-9dc0-4933-a5e1-7a72f5004728" path="/var/lib/kubelet/pods/46569cf9-9dc0-4933-a5e1-7a72f5004728/volumes" Nov 04 01:42:45.123598 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:45.123558 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:42:49.642248 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:42:49.642206 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:42:50.625602 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:50.625569 5995 scope.go:122] "RemoveContainer" containerID="5a7fa114e698b5831affaf6fa89f71cbe4b8223fd7da24312d1cd9f4fda6462f" Nov 04 01:42:54.123732 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:42:54.123694 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:42:57.951484 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:42:57.951450 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ca76f974b55b: secret "agc-cred-ca76f974b55b" not found Nov 04 01:42:57.951484 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:42:57.951476 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7789777d6-l8vds: secret "agc-cred-ca76f974b55b" not found Nov 04 01:42:57.951921 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:42:57.951531 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials podName:95384e76-af24-412c-b18e-2177d638c7d4 nodeName:}" failed. No retries permitted until 2026-11-04 01:44:59.951515429 +0000 UTC m=+2049.983248810 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials") pod "payload-processing-external-model-7789777d6-l8vds" (UID: "95384e76-af24-412c-b18e-2177d638c7d4") : secret "agc-cred-ca76f974b55b" not found Nov 04 01:43:04.123636 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:43:04.123593 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:43:14.124312 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:43:14.124270 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:43:19.640856 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:43:19.640818 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:43:20.048258 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:43:20.048226 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2"] Nov 04 01:43:20.094360 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:43:20.094311 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e01ccdf-rchd2"] Nov 04 01:43:20.130193 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:43:20.130156 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31857d37-58de-4a1b-a3bf-856b7f20c244" path="/var/lib/kubelet/pods/31857d37-58de-4a1b-a3bf-856b7f20c244/volumes" Nov 04 01:43:24.124345 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:43:24.124298 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:43:34.123694 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:43:34.123657 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:43:44.123911 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:43:44.123870 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:43:49.641746 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:43:49.641701 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:43:50.654020 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:43:50.653977 5995 scope.go:122] "RemoveContainer" containerID="7d0a61072879a14f7421be093e0f127e7d8693b6defb93daf8a6d664eb9f6868" Nov 04 01:43:50.663466 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:43:50.663444 5995 scope.go:122] "RemoveContainer" containerID="7fd25331179f4e89c2cdfdcee711194b8525f25ba60641341004656f262115f4" Nov 04 01:43:54.124387 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:43:54.124347 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:44:01.046367 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:44:01.046273 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq"] Nov 04 01:44:01.092892 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:44:01.092860 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80ca214-qchfq"] Nov 04 01:44:02.128793 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:44:02.128758 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dbdae25-2ea8-41b9-b38b-83ccda7f7a65" path="/var/lib/kubelet/pods/1dbdae25-2ea8-41b9-b38b-83ccda7f7a65/volumes" Nov 04 01:44:04.123768 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:44:04.123726 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:44:14.124275 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:44:14.124230 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:44:16.123220 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:44:16.123180 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:44:19.643027 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:44:19.642988 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:44:26.123955 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:44:26.123903 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:44:36.123341 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:44:36.123300 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:44:42.055876 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:44:42.055837 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx"] Nov 04 01:44:42.102016 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:44:42.101982 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c348c19-ptvzx"] Nov 04 01:44:42.128706 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:44:42.128676 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a6d1360-5222-47ab-83c8-8370d31dfda0" path="/var/lib/kubelet/pods/5a6d1360-5222-47ab-83c8-8370d31dfda0/volumes" Nov 04 01:44:46.123112 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:44:46.123070 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:44:49.640628 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:44:49.640590 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:44:50.691344 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:44:50.691314 5995 scope.go:122] "RemoveContainer" containerID="3b90d2464e234c608f881f2f0caf5cbaa99c64d9df559e382cc4819461ec6d36" Nov 04 01:44:50.701048 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:44:50.701024 5995 scope.go:122] "RemoveContainer" containerID="a3b680b790aa46568d8eac3fcaf842092364fda0eb3586769e5b66b6b11e3f80" Nov 04 01:44:56.123509 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:44:56.123475 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:44:58.048056 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:44:58.048018 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj"] Nov 04 01:44:58.083255 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:44:58.083219 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908e8a77-7t6gj"] Nov 04 01:44:58.128709 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:44:58.128668 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4acb3152-ac36-4c7c-8f76-84fb19118a7c" path="/var/lib/kubelet/pods/4acb3152-ac36-4c7c-8f76-84fb19118a7c/volumes" Nov 04 01:44:59.957866 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:44:59.957826 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ca76f974b55b: secret "agc-cred-ca76f974b55b" not found Nov 04 01:44:59.957866 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:44:59.957858 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7789777d6-l8vds: secret "agc-cred-ca76f974b55b" not found Nov 04 01:44:59.958462 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:44:59.957933 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials podName:95384e76-af24-412c-b18e-2177d638c7d4 nodeName:}" failed. No retries permitted until 2026-11-04 01:47:01.957914192 +0000 UTC m=+2171.989647561 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials") pod "payload-processing-external-model-7789777d6-l8vds" (UID: "95384e76-af24-412c-b18e-2177d638c7d4") : secret "agc-cred-ca76f974b55b" not found Nov 04 01:45:00.049287 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.049246 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp"] Nov 04 01:45:00.052364 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.052335 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37193fb8-rzvdp"] Nov 04 01:45:00.128708 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.128676 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0514c9e0-8b44-42a4-9d14-561aae8f2d05" path="/var/lib/kubelet/pods/0514c9e0-8b44-42a4-9d14-561aae8f2d05/volumes" Nov 04 01:45:00.179665 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.179623 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf"] Nov 04 01:45:00.180305 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.180282 5995 cpu_manager.go:414] "containerMap: removing container" podUID="9d541a44-f6eb-47ae-b9c5-244f33af1bf1" containerName="istio-proxy" Nov 04 01:45:00.180360 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.180309 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d541a44-f6eb-47ae-b9c5-244f33af1bf1" containerName="istio-proxy" Nov 04 01:45:00.180360 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.180329 5995 cpu_manager.go:414] "containerMap: removing container" podUID="12797bc0-f8ab-4176-a807-1a3918fb8d0e" containerName="istio-proxy" Nov 04 01:45:00.180360 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.180337 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12797bc0-f8ab-4176-a807-1a3918fb8d0e" containerName="istio-proxy" Nov 04 01:45:00.180360 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.180357 5995 cpu_manager.go:414] "containerMap: removing container" podUID="12797bc0-f8ab-4176-a807-1a3918fb8d0e" containerName="istio-proxy" Nov 04 01:45:00.180535 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.180365 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12797bc0-f8ab-4176-a807-1a3918fb8d0e" containerName="istio-proxy" Nov 04 01:45:00.180535 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.180374 5995 cpu_manager.go:414] "containerMap: removing container" podUID="91ff76ee-63ab-40a7-b7ef-8438908122ce" containerName="istio-proxy" Nov 04 01:45:00.180535 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.180381 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ff76ee-63ab-40a7-b7ef-8438908122ce" containerName="istio-proxy" Nov 04 01:45:00.180657 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.180540 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d541a44-f6eb-47ae-b9c5-244f33af1bf1" containerName="istio-proxy" Nov 04 01:45:00.180657 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.180557 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d541a44-f6eb-47ae-b9c5-244f33af1bf1" containerName="istio-proxy" Nov 04 01:45:00.180657 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.180567 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="12797bc0-f8ab-4176-a807-1a3918fb8d0e" containerName="istio-proxy" Nov 04 01:45:00.180657 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.180578 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="12797bc0-f8ab-4176-a807-1a3918fb8d0e" containerName="istio-proxy" Nov 04 01:45:00.180657 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.180587 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ff76ee-63ab-40a7-b7ef-8438908122ce" containerName="istio-proxy" Nov 04 01:45:00.185732 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.185711 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g"] Nov 04 01:45:00.185851 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.185831 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf" Nov 04 01:45:00.186365 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.186343 5995 cpu_manager.go:414] "containerMap: removing container" podUID="9d541a44-f6eb-47ae-b9c5-244f33af1bf1" containerName="istio-proxy" Nov 04 01:45:00.186365 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.186363 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d541a44-f6eb-47ae-b9c5-244f33af1bf1" containerName="istio-proxy" Nov 04 01:45:00.189461 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.189444 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g" Nov 04 01:45:00.235732 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.235624 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w799x\"" Nov 04 01:45:00.236433 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.236396 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:45:00.236654 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.236638 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:45:00.251501 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.251460 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf"] Nov 04 01:45:00.261222 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.261192 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4qk5\" (UniqueName: \"kubernetes.io/projected/3bf83c99-f88b-46ca-8308-7bfc6a15718d-kube-api-access-f4qk5\") pod \"maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g\" (UID: \"3bf83c99-f88b-46ca-8308-7bfc6a15718d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g" Nov 04 01:45:00.261406 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.261245 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf83c99-f88b-46ca-8308-7bfc6a15718d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g\" (UID: \"3bf83c99-f88b-46ca-8308-7bfc6a15718d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g" Nov 04 01:45:00.261406 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.261353 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knzfs\" (UniqueName: \"kubernetes.io/projected/0feb8b1c-b503-4e42-bab7-0898844f5205-kube-api-access-knzfs\") pod \"maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf\" (UID: \"0feb8b1c-b503-4e42-bab7-0898844f5205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf" Nov 04 01:45:00.261406 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.261390 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0feb8b1c-b503-4e42-bab7-0898844f5205-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf\" (UID: \"0feb8b1c-b503-4e42-bab7-0898844f5205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf" Nov 04 01:45:00.275214 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.275178 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g"] Nov 04 01:45:00.361966 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.361931 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4qk5\" (UniqueName: \"kubernetes.io/projected/3bf83c99-f88b-46ca-8308-7bfc6a15718d-kube-api-access-f4qk5\") pod \"maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g\" (UID: \"3bf83c99-f88b-46ca-8308-7bfc6a15718d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g" Nov 04 01:45:00.362181 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.361982 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf83c99-f88b-46ca-8308-7bfc6a15718d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g\" (UID: \"3bf83c99-f88b-46ca-8308-7bfc6a15718d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g" Nov 04 01:45:00.362181 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.362008 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knzfs\" (UniqueName: \"kubernetes.io/projected/0feb8b1c-b503-4e42-bab7-0898844f5205-kube-api-access-knzfs\") pod \"maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf\" (UID: \"0feb8b1c-b503-4e42-bab7-0898844f5205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf" Nov 04 01:45:00.362181 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.362046 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0feb8b1c-b503-4e42-bab7-0898844f5205-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf\" (UID: \"0feb8b1c-b503-4e42-bab7-0898844f5205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf" Nov 04 01:45:00.362798 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.362780 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0feb8b1c-b503-4e42-bab7-0898844f5205-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf\" (UID: \"0feb8b1c-b503-4e42-bab7-0898844f5205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf" Nov 04 01:45:00.362798 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.362787 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf83c99-f88b-46ca-8308-7bfc6a15718d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g\" (UID: \"3bf83c99-f88b-46ca-8308-7bfc6a15718d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g" Nov 04 01:45:00.420032 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.420001 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4qk5\" (UniqueName: \"kubernetes.io/projected/3bf83c99-f88b-46ca-8308-7bfc6a15718d-kube-api-access-f4qk5\") pod \"maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g\" (UID: \"3bf83c99-f88b-46ca-8308-7bfc6a15718d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g" Nov 04 01:45:00.420174 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.420001 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knzfs\" (UniqueName: \"kubernetes.io/projected/0feb8b1c-b503-4e42-bab7-0898844f5205-kube-api-access-knzfs\") pod \"maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf\" (UID: \"0feb8b1c-b503-4e42-bab7-0898844f5205\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf" Nov 04 01:45:00.499242 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.499215 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf" Nov 04 01:45:00.505561 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.505541 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g" Nov 04 01:45:00.872717 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.872630 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf"] Nov 04 01:45:00.879028 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:45:00.878995 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0feb8b1c_b503_4e42_bab7_0898844f5205.slice/crio-f96af6c1caae2f0d908248803662b6a6d8f36a45d2220bf81a46c7cc89077cbc WatchSource:0}: Error finding container f96af6c1caae2f0d908248803662b6a6d8f36a45d2220bf81a46c7cc89077cbc: Status 404 returned error can't find the container with id f96af6c1caae2f0d908248803662b6a6d8f36a45d2220bf81a46c7cc89077cbc Nov 04 01:45:00.924058 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:00.924027 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g"] Nov 04 01:45:00.930377 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:45:00.930346 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf83c99_f88b_46ca_8308_7bfc6a15718d.slice/crio-258cd28722f98d1a103dc36ac6095c62cd0b4d178ad388d25e99d046779c2b71 WatchSource:0}: Error finding container 258cd28722f98d1a103dc36ac6095c62cd0b4d178ad388d25e99d046779c2b71: Status 404 returned error can't find the container with id 258cd28722f98d1a103dc36ac6095c62cd0b4d178ad388d25e99d046779c2b71 Nov 04 01:45:01.022100 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:01.022064 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g" event={"ID":"3bf83c99-f88b-46ca-8308-7bfc6a15718d","Type":"ContainerStarted","Data":"258cd28722f98d1a103dc36ac6095c62cd0b4d178ad388d25e99d046779c2b71"} Nov 04 01:45:01.023486 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:01.023462 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf" event={"ID":"0feb8b1c-b503-4e42-bab7-0898844f5205","Type":"ContainerStarted","Data":"f96af6c1caae2f0d908248803662b6a6d8f36a45d2220bf81a46c7cc89077cbc"} Nov 04 01:45:02.030777 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:02.030682 5995 generic.go:356] "Generic (PLEG): container finished" podID="0feb8b1c-b503-4e42-bab7-0898844f5205" containerID="6ee217ed526137739ddef472ffae4508a51626755881039388c759a5ac8388a8" exitCode=60 Nov 04 01:45:02.030777 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:02.030724 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf" event={"ID":"0feb8b1c-b503-4e42-bab7-0898844f5205","Type":"ContainerDied","Data":"6ee217ed526137739ddef472ffae4508a51626755881039388c759a5ac8388a8"} Nov 04 01:45:02.031312 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:02.031076 5995 scope.go:122] "RemoveContainer" containerID="6ee217ed526137739ddef472ffae4508a51626755881039388c759a5ac8388a8" Nov 04 01:45:02.032604 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:02.032528 5995 generic.go:356] "Generic (PLEG): container finished" podID="3bf83c99-f88b-46ca-8308-7bfc6a15718d" containerID="183f6cdc7146410874d05d7b93e99271be49cbeb79ca79de02e43d5b5461723b" exitCode=60 Nov 04 01:45:02.032604 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:02.032588 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g" event={"ID":"3bf83c99-f88b-46ca-8308-7bfc6a15718d","Type":"ContainerDied","Data":"183f6cdc7146410874d05d7b93e99271be49cbeb79ca79de02e43d5b5461723b"} Nov 04 01:45:02.032812 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:02.032797 5995 scope.go:122] "RemoveContainer" containerID="183f6cdc7146410874d05d7b93e99271be49cbeb79ca79de02e43d5b5461723b" Nov 04 01:45:03.040191 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:03.040155 5995 generic.go:356] "Generic (PLEG): container finished" podID="3bf83c99-f88b-46ca-8308-7bfc6a15718d" containerID="0e9e92fe199d81ec46167dea9df93eaca83b5e5fbb6fcf7a545fb4ee9d2aa594" exitCode=60 Nov 04 01:45:03.040664 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:03.040239 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g" event={"ID":"3bf83c99-f88b-46ca-8308-7bfc6a15718d","Type":"ContainerDied","Data":"0e9e92fe199d81ec46167dea9df93eaca83b5e5fbb6fcf7a545fb4ee9d2aa594"} Nov 04 01:45:03.040664 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:03.040290 5995 scope.go:122] "RemoveContainer" containerID="183f6cdc7146410874d05d7b93e99271be49cbeb79ca79de02e43d5b5461723b" Nov 04 01:45:03.040664 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:03.040590 5995 scope.go:122] "RemoveContainer" containerID="0e9e92fe199d81ec46167dea9df93eaca83b5e5fbb6fcf7a545fb4ee9d2aa594" Nov 04 01:45:03.040917 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:45:03.040885 5995 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g_odh-ai-gateway-infra(3bf83c99-f88b-46ca-8308-7bfc6a15718d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g" podUID="3bf83c99-f88b-46ca-8308-7bfc6a15718d" Nov 04 01:45:03.042269 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:03.042243 5995 generic.go:356] "Generic (PLEG): container finished" podID="0feb8b1c-b503-4e42-bab7-0898844f5205" containerID="1a798db22ce8aa6ccb304d1211db5dfdeacdbe79930429209311f70ad5cc5da2" exitCode=60 Nov 04 01:45:03.042384 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:03.042296 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf" event={"ID":"0feb8b1c-b503-4e42-bab7-0898844f5205","Type":"ContainerDied","Data":"1a798db22ce8aa6ccb304d1211db5dfdeacdbe79930429209311f70ad5cc5da2"} Nov 04 01:45:03.042598 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:03.042583 5995 scope.go:122] "RemoveContainer" containerID="1a798db22ce8aa6ccb304d1211db5dfdeacdbe79930429209311f70ad5cc5da2" Nov 04 01:45:03.042826 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:45:03.042807 5995 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf_odh-ai-gateway-infra(0feb8b1c-b503-4e42-bab7-0898844f5205)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf" podUID="0feb8b1c-b503-4e42-bab7-0898844f5205" Nov 04 01:45:03.052008 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:03.051983 5995 scope.go:122] "RemoveContainer" containerID="6ee217ed526137739ddef472ffae4508a51626755881039388c759a5ac8388a8" Nov 04 01:45:04.050177 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:04.050149 5995 scope.go:122] "RemoveContainer" containerID="0e9e92fe199d81ec46167dea9df93eaca83b5e5fbb6fcf7a545fb4ee9d2aa594" Nov 04 01:45:04.050627 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:45:04.050427 5995 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g_odh-ai-gateway-infra(3bf83c99-f88b-46ca-8308-7bfc6a15718d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g" podUID="3bf83c99-f88b-46ca-8308-7bfc6a15718d" Nov 04 01:45:04.051807 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:04.051788 5995 scope.go:122] "RemoveContainer" containerID="1a798db22ce8aa6ccb304d1211db5dfdeacdbe79930429209311f70ad5cc5da2" Nov 04 01:45:04.052021 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:45:04.052002 5995 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf_odh-ai-gateway-infra(0feb8b1c-b503-4e42-bab7-0898844f5205)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf" podUID="0feb8b1c-b503-4e42-bab7-0898844f5205" Nov 04 01:45:06.123947 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:06.123913 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:45:14.122169 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:14.122137 5995 scope.go:122] "RemoveContainer" containerID="1a798db22ce8aa6ccb304d1211db5dfdeacdbe79930429209311f70ad5cc5da2" Nov 04 01:45:15.114793 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:15.114757 5995 generic.go:356] "Generic (PLEG): container finished" podID="0feb8b1c-b503-4e42-bab7-0898844f5205" containerID="8e7c64dc63aeb8412f3664e84ce8968fd869ac0dffc7f4ee32d3449e90c17801" exitCode=60 Nov 04 01:45:15.114966 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:15.114828 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf" event={"ID":"0feb8b1c-b503-4e42-bab7-0898844f5205","Type":"ContainerDied","Data":"8e7c64dc63aeb8412f3664e84ce8968fd869ac0dffc7f4ee32d3449e90c17801"} Nov 04 01:45:15.114966 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:15.114891 5995 scope.go:122] "RemoveContainer" containerID="1a798db22ce8aa6ccb304d1211db5dfdeacdbe79930429209311f70ad5cc5da2" Nov 04 01:45:15.115271 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:15.115245 5995 scope.go:122] "RemoveContainer" containerID="8e7c64dc63aeb8412f3664e84ce8968fd869ac0dffc7f4ee32d3449e90c17801" Nov 04 01:45:15.116087 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:45:15.116037 5995 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf_odh-ai-gateway-infra(0feb8b1c-b503-4e42-bab7-0898844f5205)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf" podUID="0feb8b1c-b503-4e42-bab7-0898844f5205" Nov 04 01:45:16.047647 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:16.047617 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5"] Nov 04 01:45:16.094067 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:16.094033 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90515613-r97r5"] Nov 04 01:45:16.121883 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:16.121854 5995 scope.go:122] "RemoveContainer" containerID="0e9e92fe199d81ec46167dea9df93eaca83b5e5fbb6fcf7a545fb4ee9d2aa594" Nov 04 01:45:16.123028 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:16.122993 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:45:16.128991 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:16.128966 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3934445a-0626-428a-a390-a49783fc7e73" path="/var/lib/kubelet/pods/3934445a-0626-428a-a390-a49783fc7e73/volumes" Nov 04 01:45:16.235504 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:16.235470 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf"] Nov 04 01:45:16.389663 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:16.389639 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf" Nov 04 01:45:16.552984 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:16.552949 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0feb8b1c-b503-4e42-bab7-0898844f5205-kube-api-access-knzfs\" (UniqueName: \"kubernetes.io/projected/0feb8b1c-b503-4e42-bab7-0898844f5205-kube-api-access-knzfs\") pod \"0feb8b1c-b503-4e42-bab7-0898844f5205\" (UID: \"0feb8b1c-b503-4e42-bab7-0898844f5205\") " Nov 04 01:45:16.553160 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:16.553018 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0feb8b1c-b503-4e42-bab7-0898844f5205-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0feb8b1c-b503-4e42-bab7-0898844f5205-maas-api-service-ca\") pod \"0feb8b1c-b503-4e42-bab7-0898844f5205\" (UID: \"0feb8b1c-b503-4e42-bab7-0898844f5205\") " Nov 04 01:45:16.553380 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:16.553359 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0feb8b1c-b503-4e42-bab7-0898844f5205-maas-api-service-ca" pod "0feb8b1c-b503-4e42-bab7-0898844f5205" (UID: "0feb8b1c-b503-4e42-bab7-0898844f5205"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:45:16.553539 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:16.553526 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0feb8b1c-b503-4e42-bab7-0898844f5205-maas-api-service-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:16.555058 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:16.555039 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0feb8b1c-b503-4e42-bab7-0898844f5205-kube-api-access-knzfs" pod "0feb8b1c-b503-4e42-bab7-0898844f5205" (UID: "0feb8b1c-b503-4e42-bab7-0898844f5205"). InnerVolumeSpecName "kube-api-access-knzfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:45:16.654733 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:16.654643 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knzfs\" (UniqueName: \"kubernetes.io/projected/0feb8b1c-b503-4e42-bab7-0898844f5205-kube-api-access-knzfs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:17.129090 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:17.129061 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf" Nov 04 01:45:17.129549 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:17.129066 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf" event={"ID":"0feb8b1c-b503-4e42-bab7-0898844f5205","Type":"ContainerDied","Data":"f96af6c1caae2f0d908248803662b6a6d8f36a45d2220bf81a46c7cc89077cbc"} Nov 04 01:45:17.129549 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:17.129186 5995 scope.go:122] "RemoveContainer" containerID="8e7c64dc63aeb8412f3664e84ce8968fd869ac0dffc7f4ee32d3449e90c17801" Nov 04 01:45:17.131254 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:17.131229 5995 generic.go:356] "Generic (PLEG): container finished" podID="3bf83c99-f88b-46ca-8308-7bfc6a15718d" containerID="9dfe1fa70af0c674342ebb38e033d3301ad6eb2be11d40ba8eed2cc1c840e590" exitCode=60 Nov 04 01:45:17.131433 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:17.131301 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g" event={"ID":"3bf83c99-f88b-46ca-8308-7bfc6a15718d","Type":"ContainerDied","Data":"9dfe1fa70af0c674342ebb38e033d3301ad6eb2be11d40ba8eed2cc1c840e590"} Nov 04 01:45:17.131699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:17.131683 5995 scope.go:122] "RemoveContainer" containerID="9dfe1fa70af0c674342ebb38e033d3301ad6eb2be11d40ba8eed2cc1c840e590" Nov 04 01:45:17.131934 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:45:17.131915 5995 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g_odh-ai-gateway-infra(3bf83c99-f88b-46ca-8308-7bfc6a15718d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g" podUID="3bf83c99-f88b-46ca-8308-7bfc6a15718d" Nov 04 01:45:17.139175 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:17.138986 5995 scope.go:122] "RemoveContainer" containerID="0e9e92fe199d81ec46167dea9df93eaca83b5e5fbb6fcf7a545fb4ee9d2aa594" Nov 04 01:45:17.393184 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:17.393103 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf"] Nov 04 01:45:17.397048 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:17.397023 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1225f3-29895945-qcnjf"] Nov 04 01:45:18.128473 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:18.128441 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0feb8b1c-b503-4e42-bab7-0898844f5205" path="/var/lib/kubelet/pods/0feb8b1c-b503-4e42-bab7-0898844f5205/volumes" Nov 04 01:45:18.304848 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:18.304817 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g"] Nov 04 01:45:18.446824 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:18.446798 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g" Nov 04 01:45:18.575644 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:18.574750 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf83c99-f88b-46ca-8308-7bfc6a15718d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf83c99-f88b-46ca-8308-7bfc6a15718d-maas-api-service-ca\") pod \"3bf83c99-f88b-46ca-8308-7bfc6a15718d\" (UID: \"3bf83c99-f88b-46ca-8308-7bfc6a15718d\") " Nov 04 01:45:18.575644 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:18.574865 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf83c99-f88b-46ca-8308-7bfc6a15718d-kube-api-access-f4qk5\" (UniqueName: \"kubernetes.io/projected/3bf83c99-f88b-46ca-8308-7bfc6a15718d-kube-api-access-f4qk5\") pod \"3bf83c99-f88b-46ca-8308-7bfc6a15718d\" (UID: \"3bf83c99-f88b-46ca-8308-7bfc6a15718d\") " Nov 04 01:45:18.576237 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:18.576212 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf83c99-f88b-46ca-8308-7bfc6a15718d-maas-api-service-ca" pod "3bf83c99-f88b-46ca-8308-7bfc6a15718d" (UID: "3bf83c99-f88b-46ca-8308-7bfc6a15718d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:45:18.592039 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:18.591827 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf83c99-f88b-46ca-8308-7bfc6a15718d-kube-api-access-f4qk5" pod "3bf83c99-f88b-46ca-8308-7bfc6a15718d" (UID: "3bf83c99-f88b-46ca-8308-7bfc6a15718d"). InnerVolumeSpecName "kube-api-access-f4qk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:45:18.676694 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:18.676611 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4qk5\" (UniqueName: \"kubernetes.io/projected/3bf83c99-f88b-46ca-8308-7bfc6a15718d-kube-api-access-f4qk5\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:18.676694 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:18.676652 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf83c99-f88b-46ca-8308-7bfc6a15718d-maas-api-service-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:19.154068 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:19.154039 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g" Nov 04 01:45:19.154068 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:19.154063 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g" event={"ID":"3bf83c99-f88b-46ca-8308-7bfc6a15718d","Type":"ContainerDied","Data":"258cd28722f98d1a103dc36ac6095c62cd0b4d178ad388d25e99d046779c2b71"} Nov 04 01:45:19.154302 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:19.154138 5995 scope.go:122] "RemoveContainer" containerID="9dfe1fa70af0c674342ebb38e033d3301ad6eb2be11d40ba8eed2cc1c840e590" Nov 04 01:45:19.308424 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:19.308391 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g"] Nov 04 01:45:19.355009 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:19.354971 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a96a99-29895945-rd24g"] Nov 04 01:45:19.642094 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:45:19.642056 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:45:20.129350 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:20.129315 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf83c99-f88b-46ca-8308-7bfc6a15718d" path="/var/lib/kubelet/pods/3bf83c99-f88b-46ca-8308-7bfc6a15718d/volumes" Nov 04 01:45:26.124042 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:26.124003 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:45:28.047147 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:28.047070 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2"] Nov 04 01:45:28.050131 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:28.050101 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfa368ae-spfm2"] Nov 04 01:45:28.128355 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:28.128316 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94b6da95-c060-4c58-8915-363619306374" path="/var/lib/kubelet/pods/94b6da95-c060-4c58-8915-363619306374/volumes" Nov 04 01:45:36.123107 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:36.123067 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:45:39.122835 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:39.122796 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:45:49.123268 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:49.123229 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:45:49.641691 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:45:49.641653 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:45:50.389992 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:50.389965 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f_5600ddae-56c7-4c93-afff-2900207b0819/extract/0.log" Nov 04 01:45:50.393903 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:50.393880 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f_5600ddae-56c7-4c93-afff-2900207b0819/extract/0.log" Nov 04 01:45:50.398940 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:50.398919 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:45:50.403315 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:50.403295 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:45:50.731018 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:50.730986 5995 scope.go:122] "RemoveContainer" containerID="e2f42b1c891705a3d17094f081f6ccee2e9c7ed32ef3cc338a8aaa8801349224" Nov 04 01:45:50.739914 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:50.739894 5995 scope.go:122] "RemoveContainer" containerID="d85030d19f080e5a9afcd2877b55011eebd5643873acc49ac30372bf8bff028e" Nov 04 01:45:59.122776 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:45:59.122739 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:46:09.123755 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:09.123720 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:46:19.122861 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:19.122817 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:46:19.641134 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:46:19.641101 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:46:29.122831 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:29.122796 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:46:32.856874 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:32.856842 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx"] Nov 04 01:46:32.857279 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:32.857090 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" podUID="d392036c-7164-4328-9d05-a52e2e9eec82" containerName="istio-proxy" containerID="cri-o://b614eeedcb1f460d17dd92195d4796068f833d0a43e818cfb3f40ab5ee7d759e" gracePeriod=30 Nov 04 01:46:32.904954 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:32.904919 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24"] Nov 04 01:46:32.905198 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:32.905166 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" podUID="ec4de5e6-8906-4510-ab46-f5842d546a8a" containerName="istio-proxy" containerID="cri-o://927d887a998ee346e70a4dd4dd2050c32537e8db7ab07bdb3d8a9780257eb151" gracePeriod=30 Nov 04 01:46:32.908328 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:32.908279 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9"] Nov 04 01:46:32.908876 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:32.908795 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" podUID="91871ffd-c8d2-483a-bfc4-5261b89961aa" containerName="istio-proxy" containerID="cri-o://661b3774ad155464d9fbafc85f204cdf8da8450faa3a5a44943408ff5c030189" gracePeriod=30 Nov 04 01:46:32.950031 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:32.949998 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4"] Nov 04 01:46:32.950271 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:32.950246 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" podUID="370be073-d29b-47c0-8091-bee82e0b9dbb" containerName="istio-proxy" containerID="cri-o://594cacaefc01aff34eb527ae37a8b2ff04d85495a5c1a02556a02946d8c09061" gracePeriod=30 Nov 04 01:46:36.970175 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:36.970130 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:46:36.970672 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:36.970192 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" podUID="91871ffd-c8d2-483a-bfc4-5261b89961aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:46:38.176746 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.176719 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:46:38.237149 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.237118 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:46:38.268099 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.268073 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:46:38.278656 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.278631 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:46:38.306891 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.306863 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-workload-socket\") pod \"d392036c-7164-4328-9d05-a52e2e9eec82\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " Nov 04 01:46:38.307060 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.306903 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d392036c-7164-4328-9d05-a52e2e9eec82-istio-token\" (UniqueName: \"kubernetes.io/projected/d392036c-7164-4328-9d05-a52e2e9eec82-istio-token\") pod \"d392036c-7164-4328-9d05-a52e2e9eec82\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " Nov 04 01:46:38.307060 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.306966 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-credential-socket\") pod \"d392036c-7164-4328-9d05-a52e2e9eec82\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " Nov 04 01:46:38.307060 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.307010 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-workload-certs\") pod \"d392036c-7164-4328-9d05-a52e2e9eec82\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " Nov 04 01:46:38.307213 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.307131 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-istio-envoy\") pod \"d392036c-7164-4328-9d05-a52e2e9eec82\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " Nov 04 01:46:38.307213 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.307191 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d392036c-7164-4328-9d05-a52e2e9eec82-kube-api-access-zvdt6\" (UniqueName: \"kubernetes.io/projected/d392036c-7164-4328-9d05-a52e2e9eec82-kube-api-access-zvdt6\") pod \"d392036c-7164-4328-9d05-a52e2e9eec82\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " Nov 04 01:46:38.307213 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.307198 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-workload-socket" pod "d392036c-7164-4328-9d05-a52e2e9eec82" (UID: "d392036c-7164-4328-9d05-a52e2e9eec82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.307367 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.307230 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d392036c-7164-4328-9d05-a52e2e9eec82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d392036c-7164-4328-9d05-a52e2e9eec82-istio-podinfo\") pod \"d392036c-7164-4328-9d05-a52e2e9eec82\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " Nov 04 01:46:38.307367 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.307258 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d392036c-7164-4328-9d05-a52e2e9eec82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d392036c-7164-4328-9d05-a52e2e9eec82-istiod-ca-cert\") pod \"d392036c-7164-4328-9d05-a52e2e9eec82\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " Nov 04 01:46:38.307367 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.307286 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-istio-data\") pod \"d392036c-7164-4328-9d05-a52e2e9eec82\" (UID: \"d392036c-7164-4328-9d05-a52e2e9eec82\") " Nov 04 01:46:38.307367 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.307302 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-credential-socket" pod "d392036c-7164-4328-9d05-a52e2e9eec82" (UID: "d392036c-7164-4328-9d05-a52e2e9eec82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.307367 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.307311 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-workload-certs" pod "d392036c-7164-4328-9d05-a52e2e9eec82" (UID: "d392036c-7164-4328-9d05-a52e2e9eec82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.307674 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.307629 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-istio-data" pod "d392036c-7164-4328-9d05-a52e2e9eec82" (UID: "d392036c-7164-4328-9d05-a52e2e9eec82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.307729 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.307689 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d392036c-7164-4328-9d05-a52e2e9eec82-istiod-ca-cert" pod "d392036c-7164-4328-9d05-a52e2e9eec82" (UID: "d392036c-7164-4328-9d05-a52e2e9eec82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:46:38.307826 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.307807 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.307879 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.307830 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.307879 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.307844 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.307879 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.307855 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d392036c-7164-4328-9d05-a52e2e9eec82-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.307879 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.307867 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.309283 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.309259 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-istio-envoy" pod "d392036c-7164-4328-9d05-a52e2e9eec82" (UID: "d392036c-7164-4328-9d05-a52e2e9eec82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.309374 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.309279 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d392036c-7164-4328-9d05-a52e2e9eec82-kube-api-access-zvdt6" pod "d392036c-7164-4328-9d05-a52e2e9eec82" (UID: "d392036c-7164-4328-9d05-a52e2e9eec82"). InnerVolumeSpecName "kube-api-access-zvdt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:38.309606 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.309581 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d392036c-7164-4328-9d05-a52e2e9eec82-istio-podinfo" pod "d392036c-7164-4328-9d05-a52e2e9eec82" (UID: "d392036c-7164-4328-9d05-a52e2e9eec82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:46:38.309606 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.309596 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d392036c-7164-4328-9d05-a52e2e9eec82-istio-token" pod "d392036c-7164-4328-9d05-a52e2e9eec82" (UID: "d392036c-7164-4328-9d05-a52e2e9eec82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:38.409106 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409024 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-data\") pod \"370be073-d29b-47c0-8091-bee82e0b9dbb\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " Nov 04 01:46:38.409106 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409063 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/370be073-d29b-47c0-8091-bee82e0b9dbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370be073-d29b-47c0-8091-bee82e0b9dbb-istiod-ca-cert\") pod \"370be073-d29b-47c0-8091-bee82e0b9dbb\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " Nov 04 01:46:38.409106 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409085 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-credential-socket\") pod \"91871ffd-c8d2-483a-bfc4-5261b89961aa\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " Nov 04 01:46:38.409392 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409116 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-podinfo\") pod \"370be073-d29b-47c0-8091-bee82e0b9dbb\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " Nov 04 01:46:38.409392 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409147 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-token\" (UniqueName: \"kubernetes.io/projected/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-token\") pod \"ec4de5e6-8906-4510-ab46-f5842d546a8a\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " Nov 04 01:46:38.409392 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409173 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-workload-certs\") pod \"370be073-d29b-47c0-8091-bee82e0b9dbb\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " Nov 04 01:46:38.409392 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409207 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-workload-socket\") pod \"370be073-d29b-47c0-8091-bee82e0b9dbb\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " Nov 04 01:46:38.409392 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409345 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4de5e6-8906-4510-ab46-f5842d546a8a-kube-api-access-6nmcp\" (UniqueName: \"kubernetes.io/projected/ec4de5e6-8906-4510-ab46-f5842d546a8a-kube-api-access-6nmcp\") pod \"ec4de5e6-8906-4510-ab46-f5842d546a8a\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " Nov 04 01:46:38.409392 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409363 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-data" pod "370be073-d29b-47c0-8091-bee82e0b9dbb" (UID: "370be073-d29b-47c0-8091-bee82e0b9dbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.409736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409390 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-workload-socket\") pod \"91871ffd-c8d2-483a-bfc4-5261b89961aa\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " Nov 04 01:46:38.409736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409465 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-envoy\") pod \"ec4de5e6-8906-4510-ab46-f5842d546a8a\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " Nov 04 01:46:38.409736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409505 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-workload-certs\") pod \"ec4de5e6-8906-4510-ab46-f5842d546a8a\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " Nov 04 01:46:38.409736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409523 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370be073-d29b-47c0-8091-bee82e0b9dbb-istiod-ca-cert" pod "370be073-d29b-47c0-8091-bee82e0b9dbb" (UID: "370be073-d29b-47c0-8091-bee82e0b9dbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:46:38.409736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409526 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-workload-socket" pod "370be073-d29b-47c0-8091-bee82e0b9dbb" (UID: "370be073-d29b-47c0-8091-bee82e0b9dbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.409736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409538 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91871ffd-c8d2-483a-bfc4-5261b89961aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91871ffd-c8d2-483a-bfc4-5261b89961aa-istiod-ca-cert\") pod \"91871ffd-c8d2-483a-bfc4-5261b89961aa\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " Nov 04 01:46:38.409736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409576 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91871ffd-c8d2-483a-bfc4-5261b89961aa-kube-api-access-c2d8c\" (UniqueName: \"kubernetes.io/projected/91871ffd-c8d2-483a-bfc4-5261b89961aa-kube-api-access-c2d8c\") pod \"91871ffd-c8d2-483a-bfc4-5261b89961aa\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " Nov 04 01:46:38.409736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409629 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-podinfo\") pod \"ec4de5e6-8906-4510-ab46-f5842d546a8a\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " Nov 04 01:46:38.409736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409686 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-workload-socket\") pod \"ec4de5e6-8906-4510-ab46-f5842d546a8a\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " Nov 04 01:46:38.409736 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409718 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-credential-socket\") pod \"ec4de5e6-8906-4510-ab46-f5842d546a8a\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " Nov 04 01:46:38.410241 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409740 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-workload-certs" pod "ec4de5e6-8906-4510-ab46-f5842d546a8a" (UID: "ec4de5e6-8906-4510-ab46-f5842d546a8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.410241 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409748 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-envoy\") pod \"91871ffd-c8d2-483a-bfc4-5261b89961aa\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " Nov 04 01:46:38.410241 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409780 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-podinfo\") pod \"91871ffd-c8d2-483a-bfc4-5261b89961aa\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " Nov 04 01:46:38.410241 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409822 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec4de5e6-8906-4510-ab46-f5842d546a8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4de5e6-8906-4510-ab46-f5842d546a8a-istiod-ca-cert\") pod \"ec4de5e6-8906-4510-ab46-f5842d546a8a\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " Nov 04 01:46:38.410241 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409848 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-data\") pod \"ec4de5e6-8906-4510-ab46-f5842d546a8a\" (UID: \"ec4de5e6-8906-4510-ab46-f5842d546a8a\") " Nov 04 01:46:38.410241 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409907 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-credential-socket\") pod \"370be073-d29b-47c0-8091-bee82e0b9dbb\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " Nov 04 01:46:38.410241 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409938 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-workload-certs\") pod \"91871ffd-c8d2-483a-bfc4-5261b89961aa\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " Nov 04 01:46:38.410241 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.409979 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-data\") pod \"91871ffd-c8d2-483a-bfc4-5261b89961aa\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " Nov 04 01:46:38.410241 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410009 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370be073-d29b-47c0-8091-bee82e0b9dbb-kube-api-access-rwfz2\" (UniqueName: \"kubernetes.io/projected/370be073-d29b-47c0-8091-bee82e0b9dbb-kube-api-access-rwfz2\") pod \"370be073-d29b-47c0-8091-bee82e0b9dbb\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " Nov 04 01:46:38.410241 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410048 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91871ffd-c8d2-483a-bfc4-5261b89961aa-istiod-ca-cert" pod "91871ffd-c8d2-483a-bfc4-5261b89961aa" (UID: "91871ffd-c8d2-483a-bfc4-5261b89961aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:46:38.410241 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410052 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-token\" (UniqueName: \"kubernetes.io/projected/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-token\") pod \"370be073-d29b-47c0-8091-bee82e0b9dbb\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " Nov 04 01:46:38.410241 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410117 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-envoy\") pod \"370be073-d29b-47c0-8091-bee82e0b9dbb\" (UID: \"370be073-d29b-47c0-8091-bee82e0b9dbb\") " Nov 04 01:46:38.410241 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410160 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-token\" (UniqueName: \"kubernetes.io/projected/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-token\") pod \"91871ffd-c8d2-483a-bfc4-5261b89961aa\" (UID: \"91871ffd-c8d2-483a-bfc4-5261b89961aa\") " Nov 04 01:46:38.412398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410565 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-credential-socket" pod "91871ffd-c8d2-483a-bfc4-5261b89961aa" (UID: "91871ffd-c8d2-483a-bfc4-5261b89961aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.412398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410582 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-workload-socket" pod "91871ffd-c8d2-483a-bfc4-5261b89961aa" (UID: "91871ffd-c8d2-483a-bfc4-5261b89961aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.412398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410705 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-workload-certs" pod "370be073-d29b-47c0-8091-bee82e0b9dbb" (UID: "370be073-d29b-47c0-8091-bee82e0b9dbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.412398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410778 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.412398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410800 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370be073-d29b-47c0-8091-bee82e0b9dbb-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.412398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410817 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.412398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410831 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d392036c-7164-4328-9d05-a52e2e9eec82-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.412398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410845 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.412398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410859 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.412398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410874 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.412398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410887 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91871ffd-c8d2-483a-bfc4-5261b89961aa-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.412398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410901 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d392036c-7164-4328-9d05-a52e2e9eec82-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.412398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410916 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvdt6\" (UniqueName: \"kubernetes.io/projected/d392036c-7164-4328-9d05-a52e2e9eec82-kube-api-access-zvdt6\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.412398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410921 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-credential-socket" pod "370be073-d29b-47c0-8091-bee82e0b9dbb" (UID: "370be073-d29b-47c0-8091-bee82e0b9dbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.412398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.410931 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d392036c-7164-4328-9d05-a52e2e9eec82-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.412398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.411126 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-data" pod "91871ffd-c8d2-483a-bfc4-5261b89961aa" (UID: "91871ffd-c8d2-483a-bfc4-5261b89961aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.412398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.411240 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4de5e6-8906-4510-ab46-f5842d546a8a-istiod-ca-cert" pod "ec4de5e6-8906-4510-ab46-f5842d546a8a" (UID: "ec4de5e6-8906-4510-ab46-f5842d546a8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:46:38.413283 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.411332 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-workload-certs" pod "91871ffd-c8d2-483a-bfc4-5261b89961aa" (UID: "91871ffd-c8d2-483a-bfc4-5261b89961aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.413283 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.412148 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-data" pod "ec4de5e6-8906-4510-ab46-f5842d546a8a" (UID: "ec4de5e6-8906-4510-ab46-f5842d546a8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.413283 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.412224 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-podinfo" pod "370be073-d29b-47c0-8091-bee82e0b9dbb" (UID: "370be073-d29b-47c0-8091-bee82e0b9dbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:46:38.413283 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.412451 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-envoy" pod "ec4de5e6-8906-4510-ab46-f5842d546a8a" (UID: "ec4de5e6-8906-4510-ab46-f5842d546a8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.413283 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.412748 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4de5e6-8906-4510-ab46-f5842d546a8a-kube-api-access-6nmcp" pod "ec4de5e6-8906-4510-ab46-f5842d546a8a" (UID: "ec4de5e6-8906-4510-ab46-f5842d546a8a"). InnerVolumeSpecName "kube-api-access-6nmcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:38.413283 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.413088 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-workload-socket" pod "ec4de5e6-8906-4510-ab46-f5842d546a8a" (UID: "ec4de5e6-8906-4510-ab46-f5842d546a8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.413283 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.413174 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-credential-socket" pod "ec4de5e6-8906-4510-ab46-f5842d546a8a" (UID: "ec4de5e6-8906-4510-ab46-f5842d546a8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.414824 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.414799 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-envoy" pod "370be073-d29b-47c0-8091-bee82e0b9dbb" (UID: "370be073-d29b-47c0-8091-bee82e0b9dbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.414940 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.414814 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370be073-d29b-47c0-8091-bee82e0b9dbb-kube-api-access-rwfz2" pod "370be073-d29b-47c0-8091-bee82e0b9dbb" (UID: "370be073-d29b-47c0-8091-bee82e0b9dbb"). InnerVolumeSpecName "kube-api-access-rwfz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:38.414940 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.414844 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-podinfo" pod "ec4de5e6-8906-4510-ab46-f5842d546a8a" (UID: "ec4de5e6-8906-4510-ab46-f5842d546a8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:46:38.415353 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.415056 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-token" pod "91871ffd-c8d2-483a-bfc4-5261b89961aa" (UID: "91871ffd-c8d2-483a-bfc4-5261b89961aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:38.415353 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.415090 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91871ffd-c8d2-483a-bfc4-5261b89961aa-kube-api-access-c2d8c" pod "91871ffd-c8d2-483a-bfc4-5261b89961aa" (UID: "91871ffd-c8d2-483a-bfc4-5261b89961aa"). InnerVolumeSpecName "kube-api-access-c2d8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:38.415353 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.415114 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-token" pod "370be073-d29b-47c0-8091-bee82e0b9dbb" (UID: "370be073-d29b-47c0-8091-bee82e0b9dbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:38.415353 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.415116 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-podinfo" pod "91871ffd-c8d2-483a-bfc4-5261b89961aa" (UID: "91871ffd-c8d2-483a-bfc4-5261b89961aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:46:38.415353 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.415328 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-token" pod "ec4de5e6-8906-4510-ab46-f5842d546a8a" (UID: "ec4de5e6-8906-4510-ab46-f5842d546a8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:38.416803 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.416788 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-envoy" pod "91871ffd-c8d2-483a-bfc4-5261b89961aa" (UID: "91871ffd-c8d2-483a-bfc4-5261b89961aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:38.512212 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512176 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512212 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512204 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512212 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512214 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512223 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512236 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512246 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nmcp\" (UniqueName: \"kubernetes.io/projected/ec4de5e6-8906-4510-ab46-f5842d546a8a-kube-api-access-6nmcp\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512256 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512265 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2d8c\" (UniqueName: \"kubernetes.io/projected/91871ffd-c8d2-483a-bfc4-5261b89961aa-kube-api-access-c2d8c\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512274 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512283 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512293 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512306 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512318 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512328 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4de5e6-8906-4510-ab46-f5842d546a8a-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512338 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4de5e6-8906-4510-ab46-f5842d546a8a-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512348 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370be073-d29b-47c0-8091-bee82e0b9dbb-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512381 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512388 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91871ffd-c8d2-483a-bfc4-5261b89961aa-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512397 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwfz2\" (UniqueName: \"kubernetes.io/projected/370be073-d29b-47c0-8091-bee82e0b9dbb-kube-api-access-rwfz2\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.512507 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.512405 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370be073-d29b-47c0-8091-bee82e0b9dbb-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.603010 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.602969 5995 generic.go:356] "Generic (PLEG): container finished" podID="91871ffd-c8d2-483a-bfc4-5261b89961aa" containerID="661b3774ad155464d9fbafc85f204cdf8da8450faa3a5a44943408ff5c030189" exitCode=0 Nov 04 01:46:38.603208 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.603024 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" Nov 04 01:46:38.603208 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.603037 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" event={"ID":"91871ffd-c8d2-483a-bfc4-5261b89961aa","Type":"ContainerDied","Data":"661b3774ad155464d9fbafc85f204cdf8da8450faa3a5a44943408ff5c030189"} Nov 04 01:46:38.603208 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.603097 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9" event={"ID":"91871ffd-c8d2-483a-bfc4-5261b89961aa","Type":"ContainerDied","Data":"668f4b984584ffecd81ed44a23bc1797c376bc38b65c85b1599252d7b12c24c0"} Nov 04 01:46:38.603208 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.603128 5995 scope.go:122] "RemoveContainer" containerID="661b3774ad155464d9fbafc85f204cdf8da8450faa3a5a44943408ff5c030189" Nov 04 01:46:38.604894 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.604868 5995 generic.go:356] "Generic (PLEG): container finished" podID="d392036c-7164-4328-9d05-a52e2e9eec82" containerID="b614eeedcb1f460d17dd92195d4796068f833d0a43e818cfb3f40ab5ee7d759e" exitCode=0 Nov 04 01:46:38.604996 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.604930 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" Nov 04 01:46:38.604996 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.604944 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" event={"ID":"d392036c-7164-4328-9d05-a52e2e9eec82","Type":"ContainerDied","Data":"b614eeedcb1f460d17dd92195d4796068f833d0a43e818cfb3f40ab5ee7d759e"} Nov 04 01:46:38.605144 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.605008 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx" event={"ID":"d392036c-7164-4328-9d05-a52e2e9eec82","Type":"ContainerDied","Data":"00195f85070adf87268bf50452362a6a6592c2a2bfcab7f61a43e254e83d4cd8"} Nov 04 01:46:38.606640 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.606614 5995 generic.go:356] "Generic (PLEG): container finished" podID="ec4de5e6-8906-4510-ab46-f5842d546a8a" containerID="927d887a998ee346e70a4dd4dd2050c32537e8db7ab07bdb3d8a9780257eb151" exitCode=0 Nov 04 01:46:38.606769 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.606754 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" event={"ID":"ec4de5e6-8906-4510-ab46-f5842d546a8a","Type":"ContainerDied","Data":"927d887a998ee346e70a4dd4dd2050c32537e8db7ab07bdb3d8a9780257eb151"} Nov 04 01:46:38.606810 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.606799 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" event={"ID":"ec4de5e6-8906-4510-ab46-f5842d546a8a","Type":"ContainerDied","Data":"c9268b06f1426f2b4564c00a14214a92a504f84d8ce811e33e185263ff821f7a"} Nov 04 01:46:38.606878 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.606866 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24" Nov 04 01:46:38.618937 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.618895 5995 generic.go:356] "Generic (PLEG): container finished" podID="370be073-d29b-47c0-8091-bee82e0b9dbb" containerID="594cacaefc01aff34eb527ae37a8b2ff04d85495a5c1a02556a02946d8c09061" exitCode=0 Nov 04 01:46:38.619158 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.619052 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" event={"ID":"370be073-d29b-47c0-8091-bee82e0b9dbb","Type":"ContainerDied","Data":"594cacaefc01aff34eb527ae37a8b2ff04d85495a5c1a02556a02946d8c09061"} Nov 04 01:46:38.619158 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.619146 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" event={"ID":"370be073-d29b-47c0-8091-bee82e0b9dbb","Type":"ContainerDied","Data":"a6f7503e489a3acd3169608f2d16dc1023257cf3b20e832bec1d5b3e7bcc7b7a"} Nov 04 01:46:38.620436 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.619426 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4" Nov 04 01:46:38.621894 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.621486 5995 scope.go:122] "RemoveContainer" containerID="661b3774ad155464d9fbafc85f204cdf8da8450faa3a5a44943408ff5c030189" Nov 04 01:46:38.621992 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:46:38.621955 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661b3774ad155464d9fbafc85f204cdf8da8450faa3a5a44943408ff5c030189\": container with ID starting with 661b3774ad155464d9fbafc85f204cdf8da8450faa3a5a44943408ff5c030189 not found: ID does not exist" containerID="661b3774ad155464d9fbafc85f204cdf8da8450faa3a5a44943408ff5c030189" Nov 04 01:46:38.622036 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.621983 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661b3774ad155464d9fbafc85f204cdf8da8450faa3a5a44943408ff5c030189"} err="failed to get container status \"661b3774ad155464d9fbafc85f204cdf8da8450faa3a5a44943408ff5c030189\": rpc error: code = NotFound desc = could not find container \"661b3774ad155464d9fbafc85f204cdf8da8450faa3a5a44943408ff5c030189\": container with ID starting with 661b3774ad155464d9fbafc85f204cdf8da8450faa3a5a44943408ff5c030189 not found: ID does not exist" Nov 04 01:46:38.622036 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.622001 5995 scope.go:122] "RemoveContainer" containerID="b614eeedcb1f460d17dd92195d4796068f833d0a43e818cfb3f40ab5ee7d759e" Nov 04 01:46:38.640080 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.640057 5995 scope.go:122] "RemoveContainer" containerID="b614eeedcb1f460d17dd92195d4796068f833d0a43e818cfb3f40ab5ee7d759e" Nov 04 01:46:38.640376 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:46:38.640356 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b614eeedcb1f460d17dd92195d4796068f833d0a43e818cfb3f40ab5ee7d759e\": container with ID starting with b614eeedcb1f460d17dd92195d4796068f833d0a43e818cfb3f40ab5ee7d759e not found: ID does not exist" containerID="b614eeedcb1f460d17dd92195d4796068f833d0a43e818cfb3f40ab5ee7d759e" Nov 04 01:46:38.640463 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.640388 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b614eeedcb1f460d17dd92195d4796068f833d0a43e818cfb3f40ab5ee7d759e"} err="failed to get container status \"b614eeedcb1f460d17dd92195d4796068f833d0a43e818cfb3f40ab5ee7d759e\": rpc error: code = NotFound desc = could not find container \"b614eeedcb1f460d17dd92195d4796068f833d0a43e818cfb3f40ab5ee7d759e\": container with ID starting with b614eeedcb1f460d17dd92195d4796068f833d0a43e818cfb3f40ab5ee7d759e not found: ID does not exist" Nov 04 01:46:38.640463 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.640426 5995 scope.go:122] "RemoveContainer" containerID="927d887a998ee346e70a4dd4dd2050c32537e8db7ab07bdb3d8a9780257eb151" Nov 04 01:46:38.649126 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.649102 5995 scope.go:122] "RemoveContainer" containerID="927d887a998ee346e70a4dd4dd2050c32537e8db7ab07bdb3d8a9780257eb151" Nov 04 01:46:38.649366 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:46:38.649346 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"927d887a998ee346e70a4dd4dd2050c32537e8db7ab07bdb3d8a9780257eb151\": container with ID starting with 927d887a998ee346e70a4dd4dd2050c32537e8db7ab07bdb3d8a9780257eb151 not found: ID does not exist" containerID="927d887a998ee346e70a4dd4dd2050c32537e8db7ab07bdb3d8a9780257eb151" Nov 04 01:46:38.649465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.649376 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"927d887a998ee346e70a4dd4dd2050c32537e8db7ab07bdb3d8a9780257eb151"} err="failed to get container status \"927d887a998ee346e70a4dd4dd2050c32537e8db7ab07bdb3d8a9780257eb151\": rpc error: code = NotFound desc = could not find container \"927d887a998ee346e70a4dd4dd2050c32537e8db7ab07bdb3d8a9780257eb151\": container with ID starting with 927d887a998ee346e70a4dd4dd2050c32537e8db7ab07bdb3d8a9780257eb151 not found: ID does not exist" Nov 04 01:46:38.649465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.649400 5995 scope.go:122] "RemoveContainer" containerID="594cacaefc01aff34eb527ae37a8b2ff04d85495a5c1a02556a02946d8c09061" Nov 04 01:46:38.658017 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.657997 5995 scope.go:122] "RemoveContainer" containerID="594cacaefc01aff34eb527ae37a8b2ff04d85495a5c1a02556a02946d8c09061" Nov 04 01:46:38.658296 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:46:38.658278 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"594cacaefc01aff34eb527ae37a8b2ff04d85495a5c1a02556a02946d8c09061\": container with ID starting with 594cacaefc01aff34eb527ae37a8b2ff04d85495a5c1a02556a02946d8c09061 not found: ID does not exist" containerID="594cacaefc01aff34eb527ae37a8b2ff04d85495a5c1a02556a02946d8c09061" Nov 04 01:46:38.658360 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.658302 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594cacaefc01aff34eb527ae37a8b2ff04d85495a5c1a02556a02946d8c09061"} err="failed to get container status \"594cacaefc01aff34eb527ae37a8b2ff04d85495a5c1a02556a02946d8c09061\": rpc error: code = NotFound desc = could not find container \"594cacaefc01aff34eb527ae37a8b2ff04d85495a5c1a02556a02946d8c09061\": container with ID starting with 594cacaefc01aff34eb527ae37a8b2ff04d85495a5c1a02556a02946d8c09061 not found: ID does not exist" Nov 04 01:46:38.785205 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.785173 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9"] Nov 04 01:46:38.788927 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.788901 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dpdg9"] Nov 04 01:46:38.894773 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.894744 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4"] Nov 04 01:46:38.902043 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:38.902013 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b7kv4"] Nov 04 01:46:39.005944 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:39.005910 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24"] Nov 04 01:46:39.011930 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:39.011899 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qch24"] Nov 04 01:46:39.115638 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:39.115557 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx"] Nov 04 01:46:39.119836 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:39.119807 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-449zx"] Nov 04 01:46:39.123060 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:39.123028 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:46:40.128666 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:40.128633 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="370be073-d29b-47c0-8091-bee82e0b9dbb" path="/var/lib/kubelet/pods/370be073-d29b-47c0-8091-bee82e0b9dbb/volumes" Nov 04 01:46:40.129231 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:40.129217 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91871ffd-c8d2-483a-bfc4-5261b89961aa" path="/var/lib/kubelet/pods/91871ffd-c8d2-483a-bfc4-5261b89961aa/volumes" Nov 04 01:46:40.129629 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:40.129616 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d392036c-7164-4328-9d05-a52e2e9eec82" path="/var/lib/kubelet/pods/d392036c-7164-4328-9d05-a52e2e9eec82/volumes" Nov 04 01:46:40.129992 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:40.129981 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec4de5e6-8906-4510-ab46-f5842d546a8a" path="/var/lib/kubelet/pods/ec4de5e6-8906-4510-ab46-f5842d546a8a/volumes" Nov 04 01:46:45.123483 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:45.123451 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:46:49.648237 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:46:49.648200 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:46:50.775873 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:50.775839 5995 scope.go:122] "RemoveContainer" containerID="80917cac84590f617638880741ab8638d5eb394aae54f71c85e7f5e0718ed57a" Nov 04 01:46:50.784765 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:50.784745 5995 scope.go:122] "RemoveContainer" containerID="632dd861ed0aa4150a6f5ea3b4b3e92fc3e59091bbaa1ff8d1a4249ba337be23" Nov 04 01:46:50.792641 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:50.792619 5995 scope.go:122] "RemoveContainer" containerID="77f040ceeff0918819b0d31067b7230a5afb70c7c4c3e033826dab702e9db974" Nov 04 01:46:50.800325 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:50.800308 5995 scope.go:122] "RemoveContainer" containerID="3159fe5bb43bb54f9936684344ac5e628f764e6a966682fcb692c8ebc2ba4be8" Nov 04 01:46:55.124221 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:55.124184 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:46:57.747451 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:57.747399 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-765779c7c6-wn22v"] Nov 04 01:46:57.748818 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:57.748779 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:46:57.793660 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:46:57.793624 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7789777d6-l8vds"] Nov 04 01:47:02.049713 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:47:02.049675 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-5e61507d7978: secret "agc-cred-5e61507d7978" not found Nov 04 01:47:02.049713 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:47:02.049706 5995 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ca76f974b55b: secret "agc-cred-ca76f974b55b" not found Nov 04 01:47:02.049713 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:47:02.049716 5995 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7789777d6-l8vds: [secret "agc-cred-5e61507d7978" not found, secret "agc-cred-ca76f974b55b" not found] Nov 04 01:47:02.050177 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:47:02.049769 5995 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials podName:95384e76-af24-412c-b18e-2177d638c7d4 nodeName:}" failed. No retries permitted until 2026-11-04 01:49:04.049754891 +0000 UTC m=+2294.081488273 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials") pod "payload-processing-external-model-7789777d6-l8vds" (UID: "95384e76-af24-412c-b18e-2177d638c7d4") : [secret "agc-cred-5e61507d7978" not found, secret "agc-cred-ca76f974b55b" not found] Nov 04 01:47:02.290960 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:02.290922 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:47:02.748249 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:02.748198 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" containerID="cri-o://8dc4e1fdbaded35e27deab16d1a6d39f1efd145ded6b900221727e104270ed78" gracePeriod=25 Nov 04 01:47:02.794558 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:02.794476 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" podUID="95384e76-af24-412c-b18e-2177d638c7d4" containerName="payload-processing" containerID="cri-o://1de25db9fb1c10689b90eca9b0c7b5d633e7aecb769131cd73d4eb4641dc37ee" gracePeriod=25 Nov 04 01:47:03.044800 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.044772 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" Nov 04 01:47:03.070074 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.070050 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" Nov 04 01:47:03.160295 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.160255 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cfdc4da-e36c-43f5-9491-cca19d01a097-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cfdc4da-e36c-43f5-9491-cca19d01a097-plugins-config-volume\") pod \"4cfdc4da-e36c-43f5-9491-cca19d01a097\" (UID: \"4cfdc4da-e36c-43f5-9491-cca19d01a097\") " Nov 04 01:47:03.160539 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.160313 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cfdc4da-e36c-43f5-9491-cca19d01a097-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4cfdc4da-e36c-43f5-9491-cca19d01a097-routing-overlay-volume\") pod \"4cfdc4da-e36c-43f5-9491-cca19d01a097\" (UID: \"4cfdc4da-e36c-43f5-9491-cca19d01a097\") " Nov 04 01:47:03.160539 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.160451 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cfdc4da-e36c-43f5-9491-cca19d01a097-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4cfdc4da-e36c-43f5-9491-cca19d01a097-provider-credentials\") pod \"4cfdc4da-e36c-43f5-9491-cca19d01a097\" (UID: \"4cfdc4da-e36c-43f5-9491-cca19d01a097\") " Nov 04 01:47:03.160539 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.160526 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95384e76-af24-412c-b18e-2177d638c7d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95384e76-af24-412c-b18e-2177d638c7d4-plugins-config-volume\") pod \"95384e76-af24-412c-b18e-2177d638c7d4\" (UID: \"95384e76-af24-412c-b18e-2177d638c7d4\") " Nov 04 01:47:03.160735 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.160616 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-kube-api-access-p2fzn\" (UniqueName: \"kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-kube-api-access-p2fzn\") pod \"95384e76-af24-412c-b18e-2177d638c7d4\" (UID: \"95384e76-af24-412c-b18e-2177d638c7d4\") " Nov 04 01:47:03.160735 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.160675 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials\") pod \"95384e76-af24-412c-b18e-2177d638c7d4\" (UID: \"95384e76-af24-412c-b18e-2177d638c7d4\") " Nov 04 01:47:03.160735 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.160707 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cfdc4da-e36c-43f5-9491-cca19d01a097-routing-overlay-volume" pod "4cfdc4da-e36c-43f5-9491-cca19d01a097" (UID: "4cfdc4da-e36c-43f5-9491-cca19d01a097"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:47:03.160735 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.160721 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95384e76-af24-412c-b18e-2177d638c7d4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95384e76-af24-412c-b18e-2177d638c7d4-routing-overlay-volume\") pod \"95384e76-af24-412c-b18e-2177d638c7d4\" (UID: \"95384e76-af24-412c-b18e-2177d638c7d4\") " Nov 04 01:47:03.160921 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.160761 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cfdc4da-e36c-43f5-9491-cca19d01a097-kube-api-access-fskm6\" (UniqueName: \"kubernetes.io/projected/4cfdc4da-e36c-43f5-9491-cca19d01a097-kube-api-access-fskm6\") pod \"4cfdc4da-e36c-43f5-9491-cca19d01a097\" (UID: \"4cfdc4da-e36c-43f5-9491-cca19d01a097\") " Nov 04 01:47:03.160921 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.160862 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cfdc4da-e36c-43f5-9491-cca19d01a097-plugins-config-volume" pod "4cfdc4da-e36c-43f5-9491-cca19d01a097" (UID: "4cfdc4da-e36c-43f5-9491-cca19d01a097"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:47:03.161118 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.161098 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cfdc4da-e36c-43f5-9491-cca19d01a097-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:03.161199 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.161122 5995 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4cfdc4da-e36c-43f5-9491-cca19d01a097-routing-overlay-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:03.161636 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.161599 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95384e76-af24-412c-b18e-2177d638c7d4-routing-overlay-volume" pod "95384e76-af24-412c-b18e-2177d638c7d4" (UID: "95384e76-af24-412c-b18e-2177d638c7d4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:47:03.161936 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.161903 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95384e76-af24-412c-b18e-2177d638c7d4-plugins-config-volume" pod "95384e76-af24-412c-b18e-2177d638c7d4" (UID: "95384e76-af24-412c-b18e-2177d638c7d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:47:03.163270 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.163216 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cfdc4da-e36c-43f5-9491-cca19d01a097-provider-credentials" pod "4cfdc4da-e36c-43f5-9491-cca19d01a097" (UID: "4cfdc4da-e36c-43f5-9491-cca19d01a097"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:03.163503 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.163477 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cfdc4da-e36c-43f5-9491-cca19d01a097-kube-api-access-fskm6" pod "4cfdc4da-e36c-43f5-9491-cca19d01a097" (UID: "4cfdc4da-e36c-43f5-9491-cca19d01a097"). InnerVolumeSpecName "kube-api-access-fskm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:03.163918 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.163897 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-kube-api-access-p2fzn" pod "95384e76-af24-412c-b18e-2177d638c7d4" (UID: "95384e76-af24-412c-b18e-2177d638c7d4"). InnerVolumeSpecName "kube-api-access-p2fzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:03.164008 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.163916 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials" pod "95384e76-af24-412c-b18e-2177d638c7d4" (UID: "95384e76-af24-412c-b18e-2177d638c7d4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:03.262382 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.262341 5995 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-provider-credentials\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:03.262382 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.262378 5995 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95384e76-af24-412c-b18e-2177d638c7d4-routing-overlay-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:03.262671 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.262394 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fskm6\" (UniqueName: \"kubernetes.io/projected/4cfdc4da-e36c-43f5-9491-cca19d01a097-kube-api-access-fskm6\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:03.262671 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.262407 5995 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4cfdc4da-e36c-43f5-9491-cca19d01a097-provider-credentials\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:03.262671 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.262444 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95384e76-af24-412c-b18e-2177d638c7d4-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:03.262671 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.262458 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2fzn\" (UniqueName: \"kubernetes.io/projected/95384e76-af24-412c-b18e-2177d638c7d4-kube-api-access-p2fzn\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:03.770385 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.770354 5995 generic.go:356] "Generic (PLEG): container finished" podID="95384e76-af24-412c-b18e-2177d638c7d4" containerID="1de25db9fb1c10689b90eca9b0c7b5d633e7aecb769131cd73d4eb4641dc37ee" exitCode=0 Nov 04 01:47:03.770583 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.770426 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" Nov 04 01:47:03.770583 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.770443 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" event={"ID":"95384e76-af24-412c-b18e-2177d638c7d4","Type":"ContainerDied","Data":"1de25db9fb1c10689b90eca9b0c7b5d633e7aecb769131cd73d4eb4641dc37ee"} Nov 04 01:47:03.770583 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.770508 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7789777d6-l8vds" event={"ID":"95384e76-af24-412c-b18e-2177d638c7d4","Type":"ContainerDied","Data":"b1fa9ed5c33796ede2022f74870c96a221eb537322cc9dea907680d661b4a58f"} Nov 04 01:47:03.770583 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.770537 5995 scope.go:122] "RemoveContainer" containerID="1de25db9fb1c10689b90eca9b0c7b5d633e7aecb769131cd73d4eb4641dc37ee" Nov 04 01:47:03.772043 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.772018 5995 generic.go:356] "Generic (PLEG): container finished" podID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerID="8dc4e1fdbaded35e27deab16d1a6d39f1efd145ded6b900221727e104270ed78" exitCode=0 Nov 04 01:47:03.772188 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.772080 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" event={"ID":"4cfdc4da-e36c-43f5-9491-cca19d01a097","Type":"ContainerDied","Data":"8dc4e1fdbaded35e27deab16d1a6d39f1efd145ded6b900221727e104270ed78"} Nov 04 01:47:03.772188 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.772118 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" event={"ID":"4cfdc4da-e36c-43f5-9491-cca19d01a097","Type":"ContainerDied","Data":"8a1e01ab8a58a4bf775587353f33e6064549ae4bd00f1b6c11ee3b03cd4a9313"} Nov 04 01:47:03.772188 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.772164 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-765779c7c6-wn22v" Nov 04 01:47:03.780521 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.780499 5995 scope.go:122] "RemoveContainer" containerID="1de25db9fb1c10689b90eca9b0c7b5d633e7aecb769131cd73d4eb4641dc37ee" Nov 04 01:47:03.780786 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:47:03.780767 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de25db9fb1c10689b90eca9b0c7b5d633e7aecb769131cd73d4eb4641dc37ee\": container with ID starting with 1de25db9fb1c10689b90eca9b0c7b5d633e7aecb769131cd73d4eb4641dc37ee not found: ID does not exist" containerID="1de25db9fb1c10689b90eca9b0c7b5d633e7aecb769131cd73d4eb4641dc37ee" Nov 04 01:47:03.780834 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.780796 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de25db9fb1c10689b90eca9b0c7b5d633e7aecb769131cd73d4eb4641dc37ee"} err="failed to get container status \"1de25db9fb1c10689b90eca9b0c7b5d633e7aecb769131cd73d4eb4641dc37ee\": rpc error: code = NotFound desc = could not find container \"1de25db9fb1c10689b90eca9b0c7b5d633e7aecb769131cd73d4eb4641dc37ee\": container with ID starting with 1de25db9fb1c10689b90eca9b0c7b5d633e7aecb769131cd73d4eb4641dc37ee not found: ID does not exist" Nov 04 01:47:03.780834 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.780812 5995 scope.go:122] "RemoveContainer" containerID="8dc4e1fdbaded35e27deab16d1a6d39f1efd145ded6b900221727e104270ed78" Nov 04 01:47:03.788635 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.788615 5995 scope.go:122] "RemoveContainer" containerID="8dc4e1fdbaded35e27deab16d1a6d39f1efd145ded6b900221727e104270ed78" Nov 04 01:47:03.788900 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:47:03.788883 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc4e1fdbaded35e27deab16d1a6d39f1efd145ded6b900221727e104270ed78\": container with ID starting with 8dc4e1fdbaded35e27deab16d1a6d39f1efd145ded6b900221727e104270ed78 not found: ID does not exist" containerID="8dc4e1fdbaded35e27deab16d1a6d39f1efd145ded6b900221727e104270ed78" Nov 04 01:47:03.788943 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.788907 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc4e1fdbaded35e27deab16d1a6d39f1efd145ded6b900221727e104270ed78"} err="failed to get container status \"8dc4e1fdbaded35e27deab16d1a6d39f1efd145ded6b900221727e104270ed78\": rpc error: code = NotFound desc = could not find container \"8dc4e1fdbaded35e27deab16d1a6d39f1efd145ded6b900221727e104270ed78\": container with ID starting with 8dc4e1fdbaded35e27deab16d1a6d39f1efd145ded6b900221727e104270ed78 not found: ID does not exist" Nov 04 01:47:03.925928 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.925894 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-765779c7c6-wn22v"] Nov 04 01:47:03.972325 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:03.972290 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-765779c7c6-wn22v"] Nov 04 01:47:04.077721 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:04.077634 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7789777d6-l8vds"] Nov 04 01:47:04.081959 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:04.081933 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7789777d6-l8vds"] Nov 04 01:47:04.128851 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:04.128818 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" path="/var/lib/kubelet/pods/4cfdc4da-e36c-43f5-9491-cca19d01a097/volumes" Nov 04 01:47:04.129195 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:04.129183 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95384e76-af24-412c-b18e-2177d638c7d4" path="/var/lib/kubelet/pods/95384e76-af24-412c-b18e-2177d638c7d4/volumes" Nov 04 01:47:19.640989 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:47:19.640955 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:47:33.338218 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338176 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm"] Nov 04 01:47:33.338699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338576 5995 cpu_manager.go:414] "containerMap: removing container" podUID="ec4de5e6-8906-4510-ab46-f5842d546a8a" containerName="istio-proxy" Nov 04 01:47:33.338699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338588 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4de5e6-8906-4510-ab46-f5842d546a8a" containerName="istio-proxy" Nov 04 01:47:33.338699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338608 5995 cpu_manager.go:414] "containerMap: removing container" podUID="0feb8b1c-b503-4e42-bab7-0898844f5205" containerName="cleanup" Nov 04 01:47:33.338699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338614 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0feb8b1c-b503-4e42-bab7-0898844f5205" containerName="cleanup" Nov 04 01:47:33.338699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338622 5995 cpu_manager.go:414] "containerMap: removing container" podUID="370be073-d29b-47c0-8091-bee82e0b9dbb" containerName="istio-proxy" Nov 04 01:47:33.338699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338627 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370be073-d29b-47c0-8091-bee82e0b9dbb" containerName="istio-proxy" Nov 04 01:47:33.338699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338638 5995 cpu_manager.go:414] "containerMap: removing container" podUID="3bf83c99-f88b-46ca-8308-7bfc6a15718d" containerName="cleanup" Nov 04 01:47:33.338699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338643 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf83c99-f88b-46ca-8308-7bfc6a15718d" containerName="cleanup" Nov 04 01:47:33.338699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338651 5995 cpu_manager.go:414] "containerMap: removing container" podUID="d392036c-7164-4328-9d05-a52e2e9eec82" containerName="istio-proxy" Nov 04 01:47:33.338699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338655 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d392036c-7164-4328-9d05-a52e2e9eec82" containerName="istio-proxy" Nov 04 01:47:33.338699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338665 5995 cpu_manager.go:414] "containerMap: removing container" podUID="3bf83c99-f88b-46ca-8308-7bfc6a15718d" containerName="cleanup" Nov 04 01:47:33.338699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338670 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf83c99-f88b-46ca-8308-7bfc6a15718d" containerName="cleanup" Nov 04 01:47:33.338699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338677 5995 cpu_manager.go:414] "containerMap: removing container" podUID="3bf83c99-f88b-46ca-8308-7bfc6a15718d" containerName="cleanup" Nov 04 01:47:33.338699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338682 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf83c99-f88b-46ca-8308-7bfc6a15718d" containerName="cleanup" Nov 04 01:47:33.338699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338690 5995 cpu_manager.go:414] "containerMap: removing container" podUID="91871ffd-c8d2-483a-bfc4-5261b89961aa" containerName="istio-proxy" Nov 04 01:47:33.338699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338694 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91871ffd-c8d2-483a-bfc4-5261b89961aa" containerName="istio-proxy" Nov 04 01:47:33.338699 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338704 5995 cpu_manager.go:414] "containerMap: removing container" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" Nov 04 01:47:33.339176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338709 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" Nov 04 01:47:33.339176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338719 5995 cpu_manager.go:414] "containerMap: removing container" podUID="95384e76-af24-412c-b18e-2177d638c7d4" containerName="payload-processing" Nov 04 01:47:33.339176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338724 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95384e76-af24-412c-b18e-2177d638c7d4" containerName="payload-processing" Nov 04 01:47:33.339176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338733 5995 cpu_manager.go:414] "containerMap: removing container" podUID="0feb8b1c-b503-4e42-bab7-0898844f5205" containerName="cleanup" Nov 04 01:47:33.339176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338738 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0feb8b1c-b503-4e42-bab7-0898844f5205" containerName="cleanup" Nov 04 01:47:33.339176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338796 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cfdc4da-e36c-43f5-9491-cca19d01a097" containerName="payload-processing" Nov 04 01:47:33.339176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338806 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="0feb8b1c-b503-4e42-bab7-0898844f5205" containerName="cleanup" Nov 04 01:47:33.339176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338812 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="95384e76-af24-412c-b18e-2177d638c7d4" containerName="payload-processing" Nov 04 01:47:33.339176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338819 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="0feb8b1c-b503-4e42-bab7-0898844f5205" containerName="cleanup" Nov 04 01:47:33.339176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338825 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4de5e6-8906-4510-ab46-f5842d546a8a" containerName="istio-proxy" Nov 04 01:47:33.339176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338832 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="370be073-d29b-47c0-8091-bee82e0b9dbb" containerName="istio-proxy" Nov 04 01:47:33.339176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338838 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf83c99-f88b-46ca-8308-7bfc6a15718d" containerName="cleanup" Nov 04 01:47:33.339176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338842 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="d392036c-7164-4328-9d05-a52e2e9eec82" containerName="istio-proxy" Nov 04 01:47:33.339176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338848 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf83c99-f88b-46ca-8308-7bfc6a15718d" containerName="cleanup" Nov 04 01:47:33.339176 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.338853 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="91871ffd-c8d2-483a-bfc4-5261b89961aa" containerName="istio-proxy" Nov 04 01:47:33.342215 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.342185 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.393519 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.393488 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kcfvk\"" Nov 04 01:47:33.433274 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.433239 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09c73db-08f2-4a52-8e78-b0073127fa21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.433478 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.433285 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09c73db-08f2-4a52-8e78-b0073127fa21-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.433478 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.433366 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09c73db-08f2-4a52-8e78-b0073127fa21-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.433478 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.433391 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09c73db-08f2-4a52-8e78-b0073127fa21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.433478 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.433408 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09c73db-08f2-4a52-8e78-b0073127fa21-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.433641 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.433552 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09c73db-08f2-4a52-8e78-b0073127fa21-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.433641 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.433584 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09c73db-08f2-4a52-8e78-b0073127fa21-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.433711 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.433652 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09c73db-08f2-4a52-8e78-b0073127fa21-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.433711 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.433673 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv5nq\" (UniqueName: \"kubernetes.io/projected/a09c73db-08f2-4a52-8e78-b0073127fa21-kube-api-access-zv5nq\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.442154 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.442129 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm"] Nov 04 01:47:33.534679 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.534641 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09c73db-08f2-4a52-8e78-b0073127fa21-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.534679 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.534679 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv5nq\" (UniqueName: \"kubernetes.io/projected/a09c73db-08f2-4a52-8e78-b0073127fa21-kube-api-access-zv5nq\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.534899 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.534709 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09c73db-08f2-4a52-8e78-b0073127fa21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.534899 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.534884 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09c73db-08f2-4a52-8e78-b0073127fa21-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.535002 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.534949 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09c73db-08f2-4a52-8e78-b0073127fa21-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.535058 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.535003 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09c73db-08f2-4a52-8e78-b0073127fa21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.535058 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.535029 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09c73db-08f2-4a52-8e78-b0073127fa21-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.535173 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.535089 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09c73db-08f2-4a52-8e78-b0073127fa21-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.535173 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.535119 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09c73db-08f2-4a52-8e78-b0073127fa21-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.535173 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.535121 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09c73db-08f2-4a52-8e78-b0073127fa21-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.535336 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.535268 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09c73db-08f2-4a52-8e78-b0073127fa21-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.535501 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.535475 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09c73db-08f2-4a52-8e78-b0073127fa21-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.535617 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.535518 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09c73db-08f2-4a52-8e78-b0073127fa21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.535617 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.535519 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09c73db-08f2-4a52-8e78-b0073127fa21-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.537245 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.537218 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09c73db-08f2-4a52-8e78-b0073127fa21-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.537429 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.537399 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09c73db-08f2-4a52-8e78-b0073127fa21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.592998 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.592920 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09c73db-08f2-4a52-8e78-b0073127fa21-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.593135 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.593069 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv5nq\" (UniqueName: \"kubernetes.io/projected/a09c73db-08f2-4a52-8e78-b0073127fa21-kube-api-access-zv5nq\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-8xfgm\" (UID: \"a09c73db-08f2-4a52-8e78-b0073127fa21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:33.656259 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:33.656220 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:34.082976 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:34.082933 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm"] Nov 04 01:47:34.090021 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:47:34.089994 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09c73db_08f2_4a52_8e78_b0073127fa21.slice/crio-297c0a67910cca7ae2b5571fb8e47f84a8e7d6b0bbdfad0389ce8c1093ad892f WatchSource:0}: Error finding container 297c0a67910cca7ae2b5571fb8e47f84a8e7d6b0bbdfad0389ce8c1093ad892f: Status 404 returned error can't find the container with id 297c0a67910cca7ae2b5571fb8e47f84a8e7d6b0bbdfad0389ce8c1093ad892f Nov 04 01:47:34.092061 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:34.092027 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:47:34.092163 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:34.092108 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:47:34.092163 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:34.092154 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:47:34.946898 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:34.946859 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" event={"ID":"a09c73db-08f2-4a52-8e78-b0073127fa21","Type":"ContainerStarted","Data":"594376a001d39ba73b63a28e52548e29945be35012e204a049d2652b7aebd620"} Nov 04 01:47:34.947317 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:34.946918 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" event={"ID":"a09c73db-08f2-4a52-8e78-b0073127fa21","Type":"ContainerStarted","Data":"297c0a67910cca7ae2b5571fb8e47f84a8e7d6b0bbdfad0389ce8c1093ad892f"} Nov 04 01:47:35.109944 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:35.109889 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" podStartSLOduration=2.109875137 podStartE2EDuration="2.109875137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:47:35.10850909 +0000 UTC m=+2205.140242479" watchObservedRunningTime="2026-11-04 01:47:35.109875137 +0000 UTC m=+2205.141608518" Nov 04 01:47:35.656901 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:35.656870 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:35.658537 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:35.658507 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 04 01:47:35.658808 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:35.658574 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" podUID="a09c73db-08f2-4a52-8e78-b0073127fa21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 04 01:47:36.657545 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:36.657508 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 04 01:47:36.657911 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:36.657571 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" podUID="a09c73db-08f2-4a52-8e78-b0073127fa21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 04 01:47:37.656805 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:37.656761 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 04 01:47:37.656989 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:37.656847 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" podUID="a09c73db-08f2-4a52-8e78-b0073127fa21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 04 01:47:38.657155 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:38.657116 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 04 01:47:38.657670 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:38.657170 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" podUID="a09c73db-08f2-4a52-8e78-b0073127fa21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 04 01:47:39.656853 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:39.656816 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 04 01:47:39.657040 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:39.656880 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" podUID="a09c73db-08f2-4a52-8e78-b0073127fa21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 04 01:47:39.927894 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:39.927795 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw"] Nov 04 01:47:39.928372 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:39.928054 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" podUID="ae00f727-de4d-4432-8f93-f8ad9d1aae98" containerName="istio-proxy" containerID="cri-o://960df09cfd90851126a1e45e20522c87c22025f6988a79924eca6ebbf5d36ad4" gracePeriod=30 Nov 04 01:47:40.657047 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:40.657012 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 04 01:47:40.657222 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:40.657068 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" podUID="a09c73db-08f2-4a52-8e78-b0073127fa21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 04 01:47:41.657236 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:41.657194 5995 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 04 01:47:41.657661 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:41.657276 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" podUID="a09c73db-08f2-4a52-8e78-b0073127fa21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 04 01:47:42.610303 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:42.610256 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:47:42.660872 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:42.660842 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:42.661294 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:42.661104 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:42.661920 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:42.661904 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-8xfgm" Nov 04 01:47:45.178017 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.177993 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:47:45.279038 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.278942 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-podinfo\") pod \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " Nov 04 01:47:45.279038 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.278995 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-token\" (UniqueName: \"kubernetes.io/projected/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-token\") pod \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " Nov 04 01:47:45.279038 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.279021 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae00f727-de4d-4432-8f93-f8ad9d1aae98-kube-api-access-7d7pz\" (UniqueName: \"kubernetes.io/projected/ae00f727-de4d-4432-8f93-f8ad9d1aae98-kube-api-access-7d7pz\") pod \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " Nov 04 01:47:45.279331 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.279047 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-workload-socket\") pod \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " Nov 04 01:47:45.279331 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.279079 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-credential-socket\") pod \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " Nov 04 01:47:45.279331 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.279127 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-data\") pod \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " Nov 04 01:47:45.279331 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.279154 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-envoy\") pod \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " Nov 04 01:47:45.279331 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.279211 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istiod-ca-cert\") pod \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " Nov 04 01:47:45.279331 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.279317 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-workload-certs\") pod \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\" (UID: \"ae00f727-de4d-4432-8f93-f8ad9d1aae98\") " Nov 04 01:47:45.279659 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.279352 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-workload-socket" pod "ae00f727-de4d-4432-8f93-f8ad9d1aae98" (UID: "ae00f727-de4d-4432-8f93-f8ad9d1aae98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:47:45.279659 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.279397 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-credential-socket" pod "ae00f727-de4d-4432-8f93-f8ad9d1aae98" (UID: "ae00f727-de4d-4432-8f93-f8ad9d1aae98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:47:45.279659 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.279466 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-data" pod "ae00f727-de4d-4432-8f93-f8ad9d1aae98" (UID: "ae00f727-de4d-4432-8f93-f8ad9d1aae98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:47:45.279782 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.279654 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istiod-ca-cert" pod "ae00f727-de4d-4432-8f93-f8ad9d1aae98" (UID: "ae00f727-de4d-4432-8f93-f8ad9d1aae98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:47:45.279782 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.279669 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-workload-certs" pod "ae00f727-de4d-4432-8f93-f8ad9d1aae98" (UID: "ae00f727-de4d-4432-8f93-f8ad9d1aae98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:47:45.279782 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.279686 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:45.279782 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.279706 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:45.279782 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.279721 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:45.281347 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.281328 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae00f727-de4d-4432-8f93-f8ad9d1aae98-kube-api-access-7d7pz" pod "ae00f727-de4d-4432-8f93-f8ad9d1aae98" (UID: "ae00f727-de4d-4432-8f93-f8ad9d1aae98"). InnerVolumeSpecName "kube-api-access-7d7pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:45.281590 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.281568 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-token" pod "ae00f727-de4d-4432-8f93-f8ad9d1aae98" (UID: "ae00f727-de4d-4432-8f93-f8ad9d1aae98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:45.281654 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.281576 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-envoy" pod "ae00f727-de4d-4432-8f93-f8ad9d1aae98" (UID: "ae00f727-de4d-4432-8f93-f8ad9d1aae98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:47:45.281654 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.281600 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-podinfo" pod "ae00f727-de4d-4432-8f93-f8ad9d1aae98" (UID: "ae00f727-de4d-4432-8f93-f8ad9d1aae98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:47:45.381085 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.381054 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:45.381085 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.381082 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:45.381085 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.381092 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:45.381303 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.381102 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:45.381303 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.381112 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d7pz\" (UniqueName: \"kubernetes.io/projected/ae00f727-de4d-4432-8f93-f8ad9d1aae98-kube-api-access-7d7pz\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:45.381303 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:45.381123 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae00f727-de4d-4432-8f93-f8ad9d1aae98-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:46.036701 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:46.036666 5995 generic.go:356] "Generic (PLEG): container finished" podID="ae00f727-de4d-4432-8f93-f8ad9d1aae98" containerID="960df09cfd90851126a1e45e20522c87c22025f6988a79924eca6ebbf5d36ad4" exitCode=0 Nov 04 01:47:46.036901 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:46.036792 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" Nov 04 01:47:46.036901 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:46.036808 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" event={"ID":"ae00f727-de4d-4432-8f93-f8ad9d1aae98","Type":"ContainerDied","Data":"960df09cfd90851126a1e45e20522c87c22025f6988a79924eca6ebbf5d36ad4"} Nov 04 01:47:46.036983 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:46.036901 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw" event={"ID":"ae00f727-de4d-4432-8f93-f8ad9d1aae98","Type":"ContainerDied","Data":"788998d1a96afbd601cae574b7ee08c549b0707ec67620207c6c886ff0e45282"} Nov 04 01:47:46.036983 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:46.036930 5995 scope.go:122] "RemoveContainer" containerID="960df09cfd90851126a1e45e20522c87c22025f6988a79924eca6ebbf5d36ad4" Nov 04 01:47:46.046992 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:46.046857 5995 scope.go:122] "RemoveContainer" containerID="05a3ac89aa680107b02505ac1b0ecb74d14fbae08cca7dbc260ec43ec0a24f05" Nov 04 01:47:46.055368 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:46.055353 5995 scope.go:122] "RemoveContainer" containerID="960df09cfd90851126a1e45e20522c87c22025f6988a79924eca6ebbf5d36ad4" Nov 04 01:47:46.055612 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:47:46.055592 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960df09cfd90851126a1e45e20522c87c22025f6988a79924eca6ebbf5d36ad4\": container with ID starting with 960df09cfd90851126a1e45e20522c87c22025f6988a79924eca6ebbf5d36ad4 not found: ID does not exist" containerID="960df09cfd90851126a1e45e20522c87c22025f6988a79924eca6ebbf5d36ad4" Nov 04 01:47:46.055678 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:46.055621 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960df09cfd90851126a1e45e20522c87c22025f6988a79924eca6ebbf5d36ad4"} err="failed to get container status \"960df09cfd90851126a1e45e20522c87c22025f6988a79924eca6ebbf5d36ad4\": rpc error: code = NotFound desc = could not find container \"960df09cfd90851126a1e45e20522c87c22025f6988a79924eca6ebbf5d36ad4\": container with ID starting with 960df09cfd90851126a1e45e20522c87c22025f6988a79924eca6ebbf5d36ad4 not found: ID does not exist" Nov 04 01:47:46.055678 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:46.055640 5995 scope.go:122] "RemoveContainer" containerID="05a3ac89aa680107b02505ac1b0ecb74d14fbae08cca7dbc260ec43ec0a24f05" Nov 04 01:47:46.055878 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:47:46.055859 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a3ac89aa680107b02505ac1b0ecb74d14fbae08cca7dbc260ec43ec0a24f05\": container with ID starting with 05a3ac89aa680107b02505ac1b0ecb74d14fbae08cca7dbc260ec43ec0a24f05 not found: ID does not exist" containerID="05a3ac89aa680107b02505ac1b0ecb74d14fbae08cca7dbc260ec43ec0a24f05" Nov 04 01:47:46.055913 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:46.055885 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a3ac89aa680107b02505ac1b0ecb74d14fbae08cca7dbc260ec43ec0a24f05"} err="failed to get container status \"05a3ac89aa680107b02505ac1b0ecb74d14fbae08cca7dbc260ec43ec0a24f05\": rpc error: code = NotFound desc = could not find container \"05a3ac89aa680107b02505ac1b0ecb74d14fbae08cca7dbc260ec43ec0a24f05\": container with ID starting with 05a3ac89aa680107b02505ac1b0ecb74d14fbae08cca7dbc260ec43ec0a24f05 not found: ID does not exist" Nov 04 01:47:46.191114 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:46.191085 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw"] Nov 04 01:47:46.195215 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:46.195189 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-b1047d-openshift-default-58cd5899d5-7n9xw"] Nov 04 01:47:47.451768 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:47.451733 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6"] Nov 04 01:47:47.559260 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:47.559220 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx"] Nov 04 01:47:48.128716 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:48.128682 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae00f727-de4d-4432-8f93-f8ad9d1aae98" path="/var/lib/kubelet/pods/ae00f727-de4d-4432-8f93-f8ad9d1aae98/volumes" Nov 04 01:47:49.642175 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:47:49.642145 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:47:52.452783 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:52.452734 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6" podUID="d2bd339e-06fd-46be-b1c7-8fa2792d2a7e" containerName="payload-processing" containerID="cri-o://f129b0e6584bb39462756835cd2058db00a4d638fec76d5b0eeac638f316b849" gracePeriod=25 Nov 04 01:47:52.559640 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:52.559562 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" podUID="a5ed3bab-e986-4692-a119-210933b8e3db" containerName="payload-pre-processing" containerID="cri-o://a5afce8b65604fb1711afe3440b28647a9d348558d75a3fb92ccde6de116c7c3" gracePeriod=25 Nov 04 01:47:52.709212 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:52.709154 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6" Nov 04 01:47:52.847987 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:52.847958 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2bd339e-06fd-46be-b1c7-8fa2792d2a7e-kube-api-access-9257l\" (UniqueName: \"kubernetes.io/projected/d2bd339e-06fd-46be-b1c7-8fa2792d2a7e-kube-api-access-9257l\") pod \"d2bd339e-06fd-46be-b1c7-8fa2792d2a7e\" (UID: \"d2bd339e-06fd-46be-b1c7-8fa2792d2a7e\") " Nov 04 01:47:52.848159 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:52.848056 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2bd339e-06fd-46be-b1c7-8fa2792d2a7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bd339e-06fd-46be-b1c7-8fa2792d2a7e-plugins-config-volume\") pod \"d2bd339e-06fd-46be-b1c7-8fa2792d2a7e\" (UID: \"d2bd339e-06fd-46be-b1c7-8fa2792d2a7e\") " Nov 04 01:47:52.848497 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:52.848471 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2bd339e-06fd-46be-b1c7-8fa2792d2a7e-plugins-config-volume" pod "d2bd339e-06fd-46be-b1c7-8fa2792d2a7e" (UID: "d2bd339e-06fd-46be-b1c7-8fa2792d2a7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:47:52.850055 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:52.850028 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2bd339e-06fd-46be-b1c7-8fa2792d2a7e-kube-api-access-9257l" pod "d2bd339e-06fd-46be-b1c7-8fa2792d2a7e" (UID: "d2bd339e-06fd-46be-b1c7-8fa2792d2a7e"). InnerVolumeSpecName "kube-api-access-9257l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:52.949724 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:52.949691 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9257l\" (UniqueName: \"kubernetes.io/projected/d2bd339e-06fd-46be-b1c7-8fa2792d2a7e-kube-api-access-9257l\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:52.949724 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:52.949718 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bd339e-06fd-46be-b1c7-8fa2792d2a7e-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:53.085210 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:53.085122 5995 generic.go:356] "Generic (PLEG): container finished" podID="d2bd339e-06fd-46be-b1c7-8fa2792d2a7e" containerID="f129b0e6584bb39462756835cd2058db00a4d638fec76d5b0eeac638f316b849" exitCode=0 Nov 04 01:47:53.085210 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:53.085165 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6" Nov 04 01:47:53.085472 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:53.085208 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6" event={"ID":"d2bd339e-06fd-46be-b1c7-8fa2792d2a7e","Type":"ContainerDied","Data":"f129b0e6584bb39462756835cd2058db00a4d638fec76d5b0eeac638f316b849"} Nov 04 01:47:53.085472 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:53.085268 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6" event={"ID":"d2bd339e-06fd-46be-b1c7-8fa2792d2a7e","Type":"ContainerDied","Data":"e8e442ddca31409ccaac9a071b54405f45cab9afa9a88c4cf20bfb52e6a8dfad"} Nov 04 01:47:53.085472 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:53.085295 5995 scope.go:122] "RemoveContainer" containerID="f129b0e6584bb39462756835cd2058db00a4d638fec76d5b0eeac638f316b849" Nov 04 01:47:53.094951 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:53.094935 5995 scope.go:122] "RemoveContainer" containerID="f129b0e6584bb39462756835cd2058db00a4d638fec76d5b0eeac638f316b849" Nov 04 01:47:53.095237 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:47:53.095216 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f129b0e6584bb39462756835cd2058db00a4d638fec76d5b0eeac638f316b849\": container with ID starting with f129b0e6584bb39462756835cd2058db00a4d638fec76d5b0eeac638f316b849 not found: ID does not exist" containerID="f129b0e6584bb39462756835cd2058db00a4d638fec76d5b0eeac638f316b849" Nov 04 01:47:53.095328 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:53.095244 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f129b0e6584bb39462756835cd2058db00a4d638fec76d5b0eeac638f316b849"} err="failed to get container status \"f129b0e6584bb39462756835cd2058db00a4d638fec76d5b0eeac638f316b849\": rpc error: code = NotFound desc = could not find container \"f129b0e6584bb39462756835cd2058db00a4d638fec76d5b0eeac638f316b849\": container with ID starting with f129b0e6584bb39462756835cd2058db00a4d638fec76d5b0eeac638f316b849 not found: ID does not exist" Nov 04 01:47:53.240232 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:53.240188 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6"] Nov 04 01:47:53.286573 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:53.286535 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1225f3-5b668b7888-lr2d6"] Nov 04 01:47:54.127873 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:54.127838 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2bd339e-06fd-46be-b1c7-8fa2792d2a7e" path="/var/lib/kubelet/pods/d2bd339e-06fd-46be-b1c7-8fa2792d2a7e/volumes" Nov 04 01:47:56.846025 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:47:56.845982 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" podUID="a5ed3bab-e986-4692-a119-210933b8e3db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:48:03.399717 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:03.399683 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:48:06.846153 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:06.846108 5995 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" podUID="a5ed3bab-e986-4692-a119-210933b8e3db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:48:10.504335 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.504293 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk"] Nov 04 01:48:10.504837 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.504706 5995 cpu_manager.go:414] "containerMap: removing container" podUID="ae00f727-de4d-4432-8f93-f8ad9d1aae98" containerName="istio-proxy" Nov 04 01:48:10.504837 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.504718 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae00f727-de4d-4432-8f93-f8ad9d1aae98" containerName="istio-proxy" Nov 04 01:48:10.504837 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.504738 5995 cpu_manager.go:414] "containerMap: removing container" podUID="ae00f727-de4d-4432-8f93-f8ad9d1aae98" containerName="istio-proxy" Nov 04 01:48:10.504837 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.504743 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae00f727-de4d-4432-8f93-f8ad9d1aae98" containerName="istio-proxy" Nov 04 01:48:10.504837 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.504753 5995 cpu_manager.go:414] "containerMap: removing container" podUID="d2bd339e-06fd-46be-b1c7-8fa2792d2a7e" containerName="payload-processing" Nov 04 01:48:10.504837 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.504759 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2bd339e-06fd-46be-b1c7-8fa2792d2a7e" containerName="payload-processing" Nov 04 01:48:10.504837 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.504777 5995 cpu_manager.go:414] "containerMap: removing container" podUID="0feb8b1c-b503-4e42-bab7-0898844f5205" containerName="cleanup" Nov 04 01:48:10.504837 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.504782 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0feb8b1c-b503-4e42-bab7-0898844f5205" containerName="cleanup" Nov 04 01:48:10.504837 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.504840 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2bd339e-06fd-46be-b1c7-8fa2792d2a7e" containerName="payload-processing" Nov 04 01:48:10.505104 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.504850 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="0feb8b1c-b503-4e42-bab7-0898844f5205" containerName="cleanup" Nov 04 01:48:10.505104 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.504859 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae00f727-de4d-4432-8f93-f8ad9d1aae98" containerName="istio-proxy" Nov 04 01:48:10.505104 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.504866 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf83c99-f88b-46ca-8308-7bfc6a15718d" containerName="cleanup" Nov 04 01:48:10.509607 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.509588 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk" Nov 04 01:48:10.516305 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.516254 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klrrj\" (UniqueName: \"kubernetes.io/projected/11668fdb-483c-4c4d-a5ee-8c4bbde84649-kube-api-access-klrrj\") pod \"maas-api-revoke-sub-37de0964-x92rk\" (UID: \"11668fdb-483c-4c4d-a5ee-8c4bbde84649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk" Nov 04 01:48:10.516394 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.516365 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11668fdb-483c-4c4d-a5ee-8c4bbde84649-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37de0964-x92rk\" (UID: \"11668fdb-483c-4c4d-a5ee-8c4bbde84649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk" Nov 04 01:48:10.516482 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.516456 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11668fdb-483c-4c4d-a5ee-8c4bbde84649-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37de0964-x92rk\" (UID: \"11668fdb-483c-4c4d-a5ee-8c4bbde84649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk" Nov 04 01:48:10.563147 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.563107 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk"] Nov 04 01:48:10.607850 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.607817 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w799x\"" Nov 04 01:48:10.607850 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.607843 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:48:10.608056 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.607822 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:48:10.608056 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.607817 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:48:10.617298 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.617268 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klrrj\" (UniqueName: \"kubernetes.io/projected/11668fdb-483c-4c4d-a5ee-8c4bbde84649-kube-api-access-klrrj\") pod \"maas-api-revoke-sub-37de0964-x92rk\" (UID: \"11668fdb-483c-4c4d-a5ee-8c4bbde84649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk" Nov 04 01:48:10.617512 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.617326 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11668fdb-483c-4c4d-a5ee-8c4bbde84649-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37de0964-x92rk\" (UID: \"11668fdb-483c-4c4d-a5ee-8c4bbde84649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk" Nov 04 01:48:10.617512 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.617361 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11668fdb-483c-4c4d-a5ee-8c4bbde84649-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37de0964-x92rk\" (UID: \"11668fdb-483c-4c4d-a5ee-8c4bbde84649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk" Nov 04 01:48:10.618030 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.618005 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11668fdb-483c-4c4d-a5ee-8c4bbde84649-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37de0964-x92rk\" (UID: \"11668fdb-483c-4c4d-a5ee-8c4bbde84649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk" Nov 04 01:48:10.620441 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.620390 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11668fdb-483c-4c4d-a5ee-8c4bbde84649-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37de0964-x92rk\" (UID: \"11668fdb-483c-4c4d-a5ee-8c4bbde84649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk" Nov 04 01:48:10.674998 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.674967 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klrrj\" (UniqueName: \"kubernetes.io/projected/11668fdb-483c-4c4d-a5ee-8c4bbde84649-kube-api-access-klrrj\") pod \"maas-api-revoke-sub-37de0964-x92rk\" (UID: \"11668fdb-483c-4c4d-a5ee-8c4bbde84649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk" Nov 04 01:48:10.822712 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:10.822628 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk" Nov 04 01:48:11.191799 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:11.191709 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk"] Nov 04 01:48:11.198008 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:48:11.197972 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11668fdb_483c_4c4d_a5ee_8c4bbde84649.slice/crio-6a47ed6da9974f501fb0d3f5a7140c003d573d8c79241ceae5eb42189fcd19fb WatchSource:0}: Error finding container 6a47ed6da9974f501fb0d3f5a7140c003d573d8c79241ceae5eb42189fcd19fb: Status 404 returned error can't find the container with id 6a47ed6da9974f501fb0d3f5a7140c003d573d8c79241ceae5eb42189fcd19fb Nov 04 01:48:12.185167 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:12.185135 5995 generic.go:356] "Generic (PLEG): container finished" podID="11668fdb-483c-4c4d-a5ee-8c4bbde84649" containerID="fddb040113c4f5f4c56c17cd453e5e86a33390ec8ce840b4165415dd09bbbf43" exitCode=0 Nov 04 01:48:12.185574 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:12.185204 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk" event={"ID":"11668fdb-483c-4c4d-a5ee-8c4bbde84649","Type":"ContainerDied","Data":"fddb040113c4f5f4c56c17cd453e5e86a33390ec8ce840b4165415dd09bbbf43"} Nov 04 01:48:12.185574 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:12.185241 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk" event={"ID":"11668fdb-483c-4c4d-a5ee-8c4bbde84649","Type":"ContainerStarted","Data":"6a47ed6da9974f501fb0d3f5a7140c003d573d8c79241ceae5eb42189fcd19fb"} Nov 04 01:48:12.824181 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:12.824114 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" Nov 04 01:48:12.832380 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:12.832354 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ed3bab-e986-4692-a119-210933b8e3db-kube-api-access-8pchr\" (UniqueName: \"kubernetes.io/projected/a5ed3bab-e986-4692-a119-210933b8e3db-kube-api-access-8pchr\") pod \"a5ed3bab-e986-4692-a119-210933b8e3db\" (UID: \"a5ed3bab-e986-4692-a119-210933b8e3db\") " Nov 04 01:48:12.832528 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:12.832394 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ed3bab-e986-4692-a119-210933b8e3db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ed3bab-e986-4692-a119-210933b8e3db-plugins-config-volume\") pod \"a5ed3bab-e986-4692-a119-210933b8e3db\" (UID: \"a5ed3bab-e986-4692-a119-210933b8e3db\") " Nov 04 01:48:12.832809 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:12.832786 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ed3bab-e986-4692-a119-210933b8e3db-plugins-config-volume" pod "a5ed3bab-e986-4692-a119-210933b8e3db" (UID: "a5ed3bab-e986-4692-a119-210933b8e3db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:48:12.834479 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:12.834456 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ed3bab-e986-4692-a119-210933b8e3db-kube-api-access-8pchr" pod "a5ed3bab-e986-4692-a119-210933b8e3db" (UID: "a5ed3bab-e986-4692-a119-210933b8e3db"). InnerVolumeSpecName "kube-api-access-8pchr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:48:12.933361 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:12.933324 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pchr\" (UniqueName: \"kubernetes.io/projected/a5ed3bab-e986-4692-a119-210933b8e3db-kube-api-access-8pchr\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:12.933361 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:12.933355 5995 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ed3bab-e986-4692-a119-210933b8e3db-plugins-config-volume\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:13.192011 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.191910 5995 generic.go:356] "Generic (PLEG): container finished" podID="a5ed3bab-e986-4692-a119-210933b8e3db" containerID="a5afce8b65604fb1711afe3440b28647a9d348558d75a3fb92ccde6de116c7c3" exitCode=0 Nov 04 01:48:13.192011 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.191956 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" Nov 04 01:48:13.192011 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.191986 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" event={"ID":"a5ed3bab-e986-4692-a119-210933b8e3db","Type":"ContainerDied","Data":"a5afce8b65604fb1711afe3440b28647a9d348558d75a3fb92ccde6de116c7c3"} Nov 04 01:48:13.192608 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.192039 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx" event={"ID":"a5ed3bab-e986-4692-a119-210933b8e3db","Type":"ContainerDied","Data":"971596ddd6472d3fd135bb886e987659c9fa3f05f1ec283da4693a7407b2abef"} Nov 04 01:48:13.192608 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.192070 5995 scope.go:122] "RemoveContainer" containerID="a5afce8b65604fb1711afe3440b28647a9d348558d75a3fb92ccde6de116c7c3" Nov 04 01:48:13.203028 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.203005 5995 scope.go:122] "RemoveContainer" containerID="a5afce8b65604fb1711afe3440b28647a9d348558d75a3fb92ccde6de116c7c3" Nov 04 01:48:13.203550 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:48:13.203396 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5afce8b65604fb1711afe3440b28647a9d348558d75a3fb92ccde6de116c7c3\": container with ID starting with a5afce8b65604fb1711afe3440b28647a9d348558d75a3fb92ccde6de116c7c3 not found: ID does not exist" containerID="a5afce8b65604fb1711afe3440b28647a9d348558d75a3fb92ccde6de116c7c3" Nov 04 01:48:13.203626 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.203569 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5afce8b65604fb1711afe3440b28647a9d348558d75a3fb92ccde6de116c7c3"} err="failed to get container status \"a5afce8b65604fb1711afe3440b28647a9d348558d75a3fb92ccde6de116c7c3\": rpc error: code = NotFound desc = could not find container \"a5afce8b65604fb1711afe3440b28647a9d348558d75a3fb92ccde6de116c7c3\": container with ID starting with a5afce8b65604fb1711afe3440b28647a9d348558d75a3fb92ccde6de116c7c3 not found: ID does not exist" Nov 04 01:48:13.347832 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.347779 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx"] Nov 04 01:48:13.354259 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.354233 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk" Nov 04 01:48:13.393488 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.393441 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1225f3-8494f5bf47-grtcx"] Nov 04 01:48:13.436288 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.436255 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11668fdb-483c-4c4d-a5ee-8c4bbde84649-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11668fdb-483c-4c4d-a5ee-8c4bbde84649-maas-api-openshift-ca\") pod \"11668fdb-483c-4c4d-a5ee-8c4bbde84649\" (UID: \"11668fdb-483c-4c4d-a5ee-8c4bbde84649\") " Nov 04 01:48:13.436504 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.436302 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11668fdb-483c-4c4d-a5ee-8c4bbde84649-kube-api-access-klrrj\" (UniqueName: \"kubernetes.io/projected/11668fdb-483c-4c4d-a5ee-8c4bbde84649-kube-api-access-klrrj\") pod \"11668fdb-483c-4c4d-a5ee-8c4bbde84649\" (UID: \"11668fdb-483c-4c4d-a5ee-8c4bbde84649\") " Nov 04 01:48:13.436504 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.436340 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11668fdb-483c-4c4d-a5ee-8c4bbde84649-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11668fdb-483c-4c4d-a5ee-8c4bbde84649-maas-api-xks-ca\") pod \"11668fdb-483c-4c4d-a5ee-8c4bbde84649\" (UID: \"11668fdb-483c-4c4d-a5ee-8c4bbde84649\") " Nov 04 01:48:13.436728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.436695 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11668fdb-483c-4c4d-a5ee-8c4bbde84649-maas-api-openshift-ca" pod "11668fdb-483c-4c4d-a5ee-8c4bbde84649" (UID: "11668fdb-483c-4c4d-a5ee-8c4bbde84649"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:48:13.438484 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.438459 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11668fdb-483c-4c4d-a5ee-8c4bbde84649-maas-api-xks-ca" pod "11668fdb-483c-4c4d-a5ee-8c4bbde84649" (UID: "11668fdb-483c-4c4d-a5ee-8c4bbde84649"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:48:13.438567 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.438486 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11668fdb-483c-4c4d-a5ee-8c4bbde84649-kube-api-access-klrrj" pod "11668fdb-483c-4c4d-a5ee-8c4bbde84649" (UID: "11668fdb-483c-4c4d-a5ee-8c4bbde84649"). InnerVolumeSpecName "kube-api-access-klrrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:48:13.537333 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.537291 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11668fdb-483c-4c4d-a5ee-8c4bbde84649-maas-api-openshift-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:13.537333 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.537331 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klrrj\" (UniqueName: \"kubernetes.io/projected/11668fdb-483c-4c4d-a5ee-8c4bbde84649-kube-api-access-klrrj\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:13.537598 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:13.537347 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11668fdb-483c-4c4d-a5ee-8c4bbde84649-maas-api-xks-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:14.128383 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:14.128302 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ed3bab-e986-4692-a119-210933b8e3db" path="/var/lib/kubelet/pods/a5ed3bab-e986-4692-a119-210933b8e3db/volumes" Nov 04 01:48:14.199200 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:14.199171 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk" Nov 04 01:48:14.199710 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:14.199165 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37de0964-x92rk" event={"ID":"11668fdb-483c-4c4d-a5ee-8c4bbde84649","Type":"ContainerDied","Data":"6a47ed6da9974f501fb0d3f5a7140c003d573d8c79241ceae5eb42189fcd19fb"} Nov 04 01:48:14.199710 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:14.199306 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a47ed6da9974f501fb0d3f5a7140c003d573d8c79241ceae5eb42189fcd19fb" Nov 04 01:48:14.839765 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:14.839733 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv"] Nov 04 01:48:19.641549 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:48:19.641495 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:48:21.255372 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:21.255339 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:48:23.221079 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.221042 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr"] Nov 04 01:48:23.221631 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.221610 5995 cpu_manager.go:414] "containerMap: removing container" podUID="a5ed3bab-e986-4692-a119-210933b8e3db" containerName="payload-pre-processing" Nov 04 01:48:23.221714 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.221633 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ed3bab-e986-4692-a119-210933b8e3db" containerName="payload-pre-processing" Nov 04 01:48:23.221714 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.221668 5995 cpu_manager.go:414] "containerMap: removing container" podUID="11668fdb-483c-4c4d-a5ee-8c4bbde84649" containerName="revoke-keys" Nov 04 01:48:23.221714 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.221676 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11668fdb-483c-4c4d-a5ee-8c4bbde84649" containerName="revoke-keys" Nov 04 01:48:23.221862 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.221784 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ed3bab-e986-4692-a119-210933b8e3db" containerName="payload-pre-processing" Nov 04 01:48:23.221862 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.221804 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="11668fdb-483c-4c4d-a5ee-8c4bbde84649" containerName="revoke-keys" Nov 04 01:48:23.221862 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.221815 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae00f727-de4d-4432-8f93-f8ad9d1aae98" containerName="istio-proxy" Nov 04 01:48:23.226163 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.226140 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr" Nov 04 01:48:23.276847 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.276817 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:48:23.323372 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.323336 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:48:23.323577 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.323469 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:48:23.323577 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.323544 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w799x\"" Nov 04 01:48:23.325497 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.325472 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr"] Nov 04 01:48:23.335560 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.335534 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-942af0b5-77rgr\" (UID: \"2de10d4f-f027-46c7-bfdd-ddf89c34f7dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr" Nov 04 01:48:23.335720 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.335577 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5zt7\" (UniqueName: \"kubernetes.io/projected/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-kube-api-access-x5zt7\") pod \"maas-api-revoke-sub-942af0b5-77rgr\" (UID: \"2de10d4f-f027-46c7-bfdd-ddf89c34f7dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr" Nov 04 01:48:23.335775 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.335745 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-942af0b5-77rgr\" (UID: \"2de10d4f-f027-46c7-bfdd-ddf89c34f7dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr" Nov 04 01:48:23.436511 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.436472 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-942af0b5-77rgr\" (UID: \"2de10d4f-f027-46c7-bfdd-ddf89c34f7dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr" Nov 04 01:48:23.436686 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.436642 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-942af0b5-77rgr\" (UID: \"2de10d4f-f027-46c7-bfdd-ddf89c34f7dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr" Nov 04 01:48:23.436727 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.436684 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5zt7\" (UniqueName: \"kubernetes.io/projected/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-kube-api-access-x5zt7\") pod \"maas-api-revoke-sub-942af0b5-77rgr\" (UID: \"2de10d4f-f027-46c7-bfdd-ddf89c34f7dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr" Nov 04 01:48:23.437205 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.437184 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-942af0b5-77rgr\" (UID: \"2de10d4f-f027-46c7-bfdd-ddf89c34f7dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr" Nov 04 01:48:23.438961 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.438929 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-942af0b5-77rgr\" (UID: \"2de10d4f-f027-46c7-bfdd-ddf89c34f7dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr" Nov 04 01:48:23.494607 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.494540 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5zt7\" (UniqueName: \"kubernetes.io/projected/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-kube-api-access-x5zt7\") pod \"maas-api-revoke-sub-942af0b5-77rgr\" (UID: \"2de10d4f-f027-46c7-bfdd-ddf89c34f7dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr" Nov 04 01:48:23.538599 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.538568 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr" Nov 04 01:48:23.929187 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:23.929106 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr"] Nov 04 01:48:23.935579 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:48:23.935550 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de10d4f_f027_46c7_bfdd_ddf89c34f7dc.slice/crio-7414601307dd696eaa127d9a9091548e15e869446d7b3cde18e3c95b933f2540 WatchSource:0}: Error finding container 7414601307dd696eaa127d9a9091548e15e869446d7b3cde18e3c95b933f2540: Status 404 returned error can't find the container with id 7414601307dd696eaa127d9a9091548e15e869446d7b3cde18e3c95b933f2540 Nov 04 01:48:24.247732 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:24.247699 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr" event={"ID":"2de10d4f-f027-46c7-bfdd-ddf89c34f7dc","Type":"ContainerStarted","Data":"7414601307dd696eaa127d9a9091548e15e869446d7b3cde18e3c95b933f2540"} Nov 04 01:48:25.253742 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:25.253709 5995 generic.go:356] "Generic (PLEG): container finished" podID="2de10d4f-f027-46c7-bfdd-ddf89c34f7dc" containerID="37a750c4b342241f786571ed49977fd08b6625a4bd75cf6ad0585d29bd5f934b" exitCode=0 Nov 04 01:48:25.254148 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:25.253795 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr" event={"ID":"2de10d4f-f027-46c7-bfdd-ddf89c34f7dc","Type":"ContainerDied","Data":"37a750c4b342241f786571ed49977fd08b6625a4bd75cf6ad0585d29bd5f934b"} Nov 04 01:48:26.398611 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:26.398585 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr" Nov 04 01:48:26.466264 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:26.466231 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-kube-api-access-x5zt7\" (UniqueName: \"kubernetes.io/projected/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-kube-api-access-x5zt7\") pod \"2de10d4f-f027-46c7-bfdd-ddf89c34f7dc\" (UID: \"2de10d4f-f027-46c7-bfdd-ddf89c34f7dc\") " Nov 04 01:48:26.466398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:26.466317 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-maas-api-openshift-ca\") pod \"2de10d4f-f027-46c7-bfdd-ddf89c34f7dc\" (UID: \"2de10d4f-f027-46c7-bfdd-ddf89c34f7dc\") " Nov 04 01:48:26.466398 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:26.466344 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-maas-api-xks-ca\") pod \"2de10d4f-f027-46c7-bfdd-ddf89c34f7dc\" (UID: \"2de10d4f-f027-46c7-bfdd-ddf89c34f7dc\") " Nov 04 01:48:26.466693 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:26.466666 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-maas-api-openshift-ca" pod "2de10d4f-f027-46c7-bfdd-ddf89c34f7dc" (UID: "2de10d4f-f027-46c7-bfdd-ddf89c34f7dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:48:26.468375 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:26.468355 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-maas-api-xks-ca" pod "2de10d4f-f027-46c7-bfdd-ddf89c34f7dc" (UID: "2de10d4f-f027-46c7-bfdd-ddf89c34f7dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:48:26.468452 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:26.468434 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-kube-api-access-x5zt7" pod "2de10d4f-f027-46c7-bfdd-ddf89c34f7dc" (UID: "2de10d4f-f027-46c7-bfdd-ddf89c34f7dc"). InnerVolumeSpecName "kube-api-access-x5zt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:48:26.567708 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:26.567615 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5zt7\" (UniqueName: \"kubernetes.io/projected/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-kube-api-access-x5zt7\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:26.567708 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:26.567656 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-maas-api-openshift-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:26.567708 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:26.567666 5995 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2de10d4f-f027-46c7-bfdd-ddf89c34f7dc-maas-api-xks-ca\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:27.266630 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:27.266603 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr" Nov 04 01:48:27.266810 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:27.266607 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-942af0b5-77rgr" event={"ID":"2de10d4f-f027-46c7-bfdd-ddf89c34f7dc","Type":"ContainerDied","Data":"7414601307dd696eaa127d9a9091548e15e869446d7b3cde18e3c95b933f2540"} Nov 04 01:48:27.266810 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:27.266725 5995 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7414601307dd696eaa127d9a9091548e15e869446d7b3cde18e3c95b933f2540" Nov 04 01:48:29.859778 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:29.859662 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" podUID="dfd73b41-d89f-4391-a0aa-1246b07c718c" containerName="main" containerID="cri-o://aa3c715a796d4afe82daea8432933ef20add3829b35b23701fea741ebce1790d" gracePeriod=45 Nov 04 01:48:30.116066 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.116001 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:48:30.205941 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.205911 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-dshm\") pod \"dfd73b41-d89f-4391-a0aa-1246b07c718c\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " Nov 04 01:48:30.206160 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.205960 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-home\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-home\") pod \"dfd73b41-d89f-4391-a0aa-1246b07c718c\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " Nov 04 01:48:30.206160 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.205996 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd73b41-d89f-4391-a0aa-1246b07c718c-tls-certs\" (UniqueName: \"kubernetes.io/secret/dfd73b41-d89f-4391-a0aa-1246b07c718c-tls-certs\") pod \"dfd73b41-d89f-4391-a0aa-1246b07c718c\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " Nov 04 01:48:30.206160 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.206096 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd73b41-d89f-4391-a0aa-1246b07c718c-kube-api-access-mnks2\" (UniqueName: \"kubernetes.io/projected/dfd73b41-d89f-4391-a0aa-1246b07c718c-kube-api-access-mnks2\") pod \"dfd73b41-d89f-4391-a0aa-1246b07c718c\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " Nov 04 01:48:30.206289 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.206269 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-tmp-dir\") pod \"dfd73b41-d89f-4391-a0aa-1246b07c718c\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " Nov 04 01:48:30.206326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.206304 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-home" pod "dfd73b41-d89f-4391-a0aa-1246b07c718c" (UID: "dfd73b41-d89f-4391-a0aa-1246b07c718c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:30.206326 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.206314 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-model-cache\") pod \"dfd73b41-d89f-4391-a0aa-1246b07c718c\" (UID: \"dfd73b41-d89f-4391-a0aa-1246b07c718c\") " Nov 04 01:48:30.206572 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.206545 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-tmp-dir" pod "dfd73b41-d89f-4391-a0aa-1246b07c718c" (UID: "dfd73b41-d89f-4391-a0aa-1246b07c718c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:30.206667 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.206644 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-model-cache" pod "dfd73b41-d89f-4391-a0aa-1246b07c718c" (UID: "dfd73b41-d89f-4391-a0aa-1246b07c718c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:30.206753 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.206730 5995 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-tmp-dir\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:30.206753 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.206753 5995 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-model-cache\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:30.206898 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.206769 5995 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-home\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:30.208340 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.208317 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-dshm" pod "dfd73b41-d89f-4391-a0aa-1246b07c718c" (UID: "dfd73b41-d89f-4391-a0aa-1246b07c718c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:30.208340 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.208328 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd73b41-d89f-4391-a0aa-1246b07c718c-tls-certs" pod "dfd73b41-d89f-4391-a0aa-1246b07c718c" (UID: "dfd73b41-d89f-4391-a0aa-1246b07c718c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:48:30.208481 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.208400 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd73b41-d89f-4391-a0aa-1246b07c718c-kube-api-access-mnks2" pod "dfd73b41-d89f-4391-a0aa-1246b07c718c" (UID: "dfd73b41-d89f-4391-a0aa-1246b07c718c"). InnerVolumeSpecName "kube-api-access-mnks2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:48:30.284971 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.284940 5995 generic.go:356] "Generic (PLEG): container finished" podID="dfd73b41-d89f-4391-a0aa-1246b07c718c" containerID="aa3c715a796d4afe82daea8432933ef20add3829b35b23701fea741ebce1790d" exitCode=0 Nov 04 01:48:30.285139 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.284981 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" Nov 04 01:48:30.285139 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.285030 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" event={"ID":"dfd73b41-d89f-4391-a0aa-1246b07c718c","Type":"ContainerDied","Data":"aa3c715a796d4afe82daea8432933ef20add3829b35b23701fea741ebce1790d"} Nov 04 01:48:30.285139 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.285073 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv" event={"ID":"dfd73b41-d89f-4391-a0aa-1246b07c718c","Type":"ContainerDied","Data":"3ef2ac342122a6fc10a5d3b5ec732f9102c62c3909e3ebd56a5497d7fb91e6f5"} Nov 04 01:48:30.285139 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.285090 5995 scope.go:122] "RemoveContainer" containerID="aa3c715a796d4afe82daea8432933ef20add3829b35b23701fea741ebce1790d" Nov 04 01:48:30.298371 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.298349 5995 scope.go:122] "RemoveContainer" containerID="aa3c715a796d4afe82daea8432933ef20add3829b35b23701fea741ebce1790d" Nov 04 01:48:30.298922 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:48:30.298885 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa3c715a796d4afe82daea8432933ef20add3829b35b23701fea741ebce1790d\": container with ID starting with aa3c715a796d4afe82daea8432933ef20add3829b35b23701fea741ebce1790d not found: ID does not exist" containerID="aa3c715a796d4afe82daea8432933ef20add3829b35b23701fea741ebce1790d" Nov 04 01:48:30.299048 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.298931 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3c715a796d4afe82daea8432933ef20add3829b35b23701fea741ebce1790d"} err="failed to get container status \"aa3c715a796d4afe82daea8432933ef20add3829b35b23701fea741ebce1790d\": rpc error: code = NotFound desc = could not find container \"aa3c715a796d4afe82daea8432933ef20add3829b35b23701fea741ebce1790d\": container with ID starting with aa3c715a796d4afe82daea8432933ef20add3829b35b23701fea741ebce1790d not found: ID does not exist" Nov 04 01:48:30.307691 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.307659 5995 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfd73b41-d89f-4391-a0aa-1246b07c718c-dshm\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:30.307791 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.307695 5995 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfd73b41-d89f-4391-a0aa-1246b07c718c-tls-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:30.307791 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.307712 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnks2\" (UniqueName: \"kubernetes.io/projected/dfd73b41-d89f-4391-a0aa-1246b07c718c-kube-api-access-mnks2\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:30.405978 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.405898 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv"] Nov 04 01:48:30.410323 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:30.410295 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a96a99/premium-w0-a96a99-kserve-7d74dc8c6f-kzqmv"] Nov 04 01:48:32.128316 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:32.128282 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd73b41-d89f-4391-a0aa-1246b07c718c" path="/var/lib/kubelet/pods/dfd73b41-d89f-4391-a0aa-1246b07c718c/volumes" Nov 04 01:48:43.698627 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:43.698593 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jf7b5"] Nov 04 01:48:44.196438 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.196395 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p"] Nov 04 01:48:44.196876 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.196859 5995 cpu_manager.go:414] "containerMap: removing container" podUID="dfd73b41-d89f-4391-a0aa-1246b07c718c" containerName="main" Nov 04 01:48:44.196967 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.196879 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd73b41-d89f-4391-a0aa-1246b07c718c" containerName="main" Nov 04 01:48:44.196967 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.196905 5995 cpu_manager.go:414] "containerMap: removing container" podUID="2de10d4f-f027-46c7-bfdd-ddf89c34f7dc" containerName="revoke-keys" Nov 04 01:48:44.196967 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.196913 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de10d4f-f027-46c7-bfdd-ddf89c34f7dc" containerName="revoke-keys" Nov 04 01:48:44.197125 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.197028 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd73b41-d89f-4391-a0aa-1246b07c718c" containerName="main" Nov 04 01:48:44.197125 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.197044 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de10d4f-f027-46c7-bfdd-ddf89c34f7dc" containerName="revoke-keys" Nov 04 01:48:44.200380 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.200356 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.251405 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.251376 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a96a99-openshift-default-dockercfg-hbmfv\"" Nov 04 01:48:44.299918 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.299884 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p"] Nov 04 01:48:44.337151 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.337113 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-envoy\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.337330 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.337165 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-podinfo\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.337330 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.337231 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6476dc98-8c94-4c02-afb2-c47c54263e00-istiod-ca-cert\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.337330 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.337254 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-token\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.337330 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.337285 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-credential-socket\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.337330 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.337305 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-data\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.337584 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.337381 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swggl\" (UniqueName: \"kubernetes.io/projected/6476dc98-8c94-4c02-afb2-c47c54263e00-kube-api-access-swggl\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.337584 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.337467 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-workload-socket\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.337584 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.337492 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-workload-certs\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.438877 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.438844 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-podinfo\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.438877 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.438878 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6476dc98-8c94-4c02-afb2-c47c54263e00-istiod-ca-cert\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.439148 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.438893 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-token\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.439148 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.438923 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-credential-socket\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.439148 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.439031 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-data\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.439148 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.439078 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swggl\" (UniqueName: \"kubernetes.io/projected/6476dc98-8c94-4c02-afb2-c47c54263e00-kube-api-access-swggl\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.439148 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.439111 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-workload-socket\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.439148 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.439135 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-workload-certs\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.439465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.439268 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-envoy\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.439465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.439324 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-credential-socket\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.439465 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.439396 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-data\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.439634 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.439516 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-workload-socket\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.439634 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.439570 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-workload-certs\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.439920 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.439901 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6476dc98-8c94-4c02-afb2-c47c54263e00-istiod-ca-cert\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.441333 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.441312 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-envoy\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.441333 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.441330 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-podinfo\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.496923 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.496855 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-token\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.497086 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.497062 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swggl\" (UniqueName: \"kubernetes.io/projected/6476dc98-8c94-4c02-afb2-c47c54263e00-kube-api-access-swggl\") pod \"e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.513763 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.513736 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:44.938530 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.938444 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p"] Nov 04 01:48:44.944990 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:48:44.944959 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6476dc98_8c94_4c02_afb2_c47c54263e00.slice/crio-7b15cf2097335690ab894f6684596c233cf92b730844b4062e0020723ac54bf2 WatchSource:0}: Error finding container 7b15cf2097335690ab894f6684596c233cf92b730844b4062e0020723ac54bf2: Status 404 returned error can't find the container with id 7b15cf2097335690ab894f6684596c233cf92b730844b4062e0020723ac54bf2 Nov 04 01:48:44.947530 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.947497 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:48:44.947620 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.947564 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:48:44.947620 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:44.947592 5995 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 04 01:48:45.367957 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:45.367920 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" event={"ID":"6476dc98-8c94-4c02-afb2-c47c54263e00","Type":"ContainerStarted","Data":"eaad778315702b3ca06a8e36988569b83b7907f92c7a2f4172fca18de90531de"} Nov 04 01:48:45.368140 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:45.367984 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" event={"ID":"6476dc98-8c94-4c02-afb2-c47c54263e00","Type":"ContainerStarted","Data":"7b15cf2097335690ab894f6684596c233cf92b730844b4062e0020723ac54bf2"} Nov 04 01:48:45.485485 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:45.485400 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" podStartSLOduration=1.4853783219999999 podStartE2EDuration="1.485378322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:48:45.483787003 +0000 UTC m=+2275.515520394" watchObservedRunningTime="2026-11-04 01:48:45.485378322 +0000 UTC m=+2275.517111714" Nov 04 01:48:45.514209 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:45.514181 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:46.515996 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:46.515954 5995 patch_prober.go:29] interesting pod/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:48:46.516391 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:46.516010 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" podUID="6476dc98-8c94-4c02-afb2-c47c54263e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 04 01:48:47.514277 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:47.514231 5995 patch_prober.go:29] interesting pod/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:48:47.514489 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:47.514284 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" podUID="6476dc98-8c94-4c02-afb2-c47c54263e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 04 01:48:48.514644 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:48.514595 5995 patch_prober.go:29] interesting pod/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:48:48.515045 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:48.514672 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" podUID="6476dc98-8c94-4c02-afb2-c47c54263e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 04 01:48:49.514703 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:49.514658 5995 patch_prober.go:29] interesting pod/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:48:49.515228 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:49.514719 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" podUID="6476dc98-8c94-4c02-afb2-c47c54263e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 04 01:48:49.643900 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:48:49.643864 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:48:50.514382 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:50.514347 5995 patch_prober.go:29] interesting pod/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:48:50.514584 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:50.514442 5995 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" podUID="6476dc98-8c94-4c02-afb2-c47c54263e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 04 01:48:51.518531 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:51.518499 5995 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:51.518924 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:51.518853 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:51.519637 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:51.519612 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:48:55.814766 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:55.814727 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p"] Nov 04 01:48:55.815202 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:55.814956 5995 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" podUID="6476dc98-8c94-4c02-afb2-c47c54263e00" containerName="istio-proxy" containerID="cri-o://eaad778315702b3ca06a8e36988569b83b7907f92c7a2f4172fca18de90531de" gracePeriod=30 Nov 04 01:48:57.902047 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:57.902021 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6qnlj_5062a83f-5161-447e-a161-72f1422286d2/manager/0.log" Nov 04 01:48:58.141777 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:58.141747 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pbz49_4e005554-58f5-4e78-9611-043b8d3ccd6a/kuadrant-console-plugin/0.log" Nov 04 01:48:59.141864 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:59.141830 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lzpmm_c282fd58-c58f-4a46-9e63-ab07b0eb2069/manager/0.log" Nov 04 01:48:59.372994 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:48:59.372956 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jf7b5_d1825e94-4a7d-4ea5-ad61-b1b4fa486373/limitador/0.log" Nov 04 01:49:01.078511 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.078481 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:49:01.103546 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.103482 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-envoy\") pod \"6476dc98-8c94-4c02-afb2-c47c54263e00\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " Nov 04 01:49:01.103546 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.103525 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6476dc98-8c94-4c02-afb2-c47c54263e00-kube-api-access-swggl\" (UniqueName: \"kubernetes.io/projected/6476dc98-8c94-4c02-afb2-c47c54263e00-kube-api-access-swggl\") pod \"6476dc98-8c94-4c02-afb2-c47c54263e00\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " Nov 04 01:49:01.103750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.103588 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-workload-certs\") pod \"6476dc98-8c94-4c02-afb2-c47c54263e00\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " Nov 04 01:49:01.103750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.103628 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6476dc98-8c94-4c02-afb2-c47c54263e00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6476dc98-8c94-4c02-afb2-c47c54263e00-istiod-ca-cert\") pod \"6476dc98-8c94-4c02-afb2-c47c54263e00\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " Nov 04 01:49:01.103750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.103658 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-podinfo\") pod \"6476dc98-8c94-4c02-afb2-c47c54263e00\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " Nov 04 01:49:01.103750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.103678 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-token\" (UniqueName: \"kubernetes.io/projected/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-token\") pod \"6476dc98-8c94-4c02-afb2-c47c54263e00\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " Nov 04 01:49:01.103750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.103699 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-credential-socket\") pod \"6476dc98-8c94-4c02-afb2-c47c54263e00\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " Nov 04 01:49:01.103750 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.103736 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-data\") pod \"6476dc98-8c94-4c02-afb2-c47c54263e00\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " Nov 04 01:49:01.104067 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.103766 5995 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-workload-socket\") pod \"6476dc98-8c94-4c02-afb2-c47c54263e00\" (UID: \"6476dc98-8c94-4c02-afb2-c47c54263e00\") " Nov 04 01:49:01.104165 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.104136 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6476dc98-8c94-4c02-afb2-c47c54263e00-istiod-ca-cert" pod "6476dc98-8c94-4c02-afb2-c47c54263e00" (UID: "6476dc98-8c94-4c02-afb2-c47c54263e00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:49:01.104280 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.104258 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-workload-socket" pod "6476dc98-8c94-4c02-afb2-c47c54263e00" (UID: "6476dc98-8c94-4c02-afb2-c47c54263e00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:49:01.104338 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.104289 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-workload-certs" pod "6476dc98-8c94-4c02-afb2-c47c54263e00" (UID: "6476dc98-8c94-4c02-afb2-c47c54263e00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:49:01.104505 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.104470 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-data" pod "6476dc98-8c94-4c02-afb2-c47c54263e00" (UID: "6476dc98-8c94-4c02-afb2-c47c54263e00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:49:01.104596 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.104508 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-credential-socket" pod "6476dc98-8c94-4c02-afb2-c47c54263e00" (UID: "6476dc98-8c94-4c02-afb2-c47c54263e00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:49:01.105771 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.105750 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-envoy" pod "6476dc98-8c94-4c02-afb2-c47c54263e00" (UID: "6476dc98-8c94-4c02-afb2-c47c54263e00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:49:01.105859 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.105774 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6476dc98-8c94-4c02-afb2-c47c54263e00-kube-api-access-swggl" pod "6476dc98-8c94-4c02-afb2-c47c54263e00" (UID: "6476dc98-8c94-4c02-afb2-c47c54263e00"). InnerVolumeSpecName "kube-api-access-swggl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:49:01.105859 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.105842 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-token" pod "6476dc98-8c94-4c02-afb2-c47c54263e00" (UID: "6476dc98-8c94-4c02-afb2-c47c54263e00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:49:01.105984 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.105966 5995 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-podinfo" pod "6476dc98-8c94-4c02-afb2-c47c54263e00" (UID: "6476dc98-8c94-4c02-afb2-c47c54263e00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:49:01.205369 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.205336 5995 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6476dc98-8c94-4c02-afb2-c47c54263e00-istiod-ca-cert\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:01.205369 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.205367 5995 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-podinfo\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:01.205369 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.205376 5995 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-token\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:01.205634 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.205385 5995 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-credential-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:01.205634 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.205397 5995 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-data\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:01.205634 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.205435 5995 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-workload-socket\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:01.205634 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.205444 5995 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-istio-envoy\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:01.205634 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.205453 5995 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swggl\" (UniqueName: \"kubernetes.io/projected/6476dc98-8c94-4c02-afb2-c47c54263e00-kube-api-access-swggl\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:01.205634 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.205462 5995 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6476dc98-8c94-4c02-afb2-c47c54263e00-workload-certs\") on node \"ip-10-0-138-179.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:01.472012 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.471975 5995 generic.go:356] "Generic (PLEG): container finished" podID="6476dc98-8c94-4c02-afb2-c47c54263e00" containerID="eaad778315702b3ca06a8e36988569b83b7907f92c7a2f4172fca18de90531de" exitCode=0 Nov 04 01:49:01.472197 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.472026 5995 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" Nov 04 01:49:01.472197 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.472061 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" event={"ID":"6476dc98-8c94-4c02-afb2-c47c54263e00","Type":"ContainerDied","Data":"eaad778315702b3ca06a8e36988569b83b7907f92c7a2f4172fca18de90531de"} Nov 04 01:49:01.472197 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.472129 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p" event={"ID":"6476dc98-8c94-4c02-afb2-c47c54263e00","Type":"ContainerDied","Data":"7b15cf2097335690ab894f6684596c233cf92b730844b4062e0020723ac54bf2"} Nov 04 01:49:01.472197 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.472159 5995 scope.go:122] "RemoveContainer" containerID="eaad778315702b3ca06a8e36988569b83b7907f92c7a2f4172fca18de90531de" Nov 04 01:49:01.483438 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.483394 5995 scope.go:122] "RemoveContainer" containerID="eaad778315702b3ca06a8e36988569b83b7907f92c7a2f4172fca18de90531de" Nov 04 01:49:01.483761 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:49:01.483743 5995 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaad778315702b3ca06a8e36988569b83b7907f92c7a2f4172fca18de90531de\": container with ID starting with eaad778315702b3ca06a8e36988569b83b7907f92c7a2f4172fca18de90531de not found: ID does not exist" containerID="eaad778315702b3ca06a8e36988569b83b7907f92c7a2f4172fca18de90531de" Nov 04 01:49:01.483835 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.483772 5995 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaad778315702b3ca06a8e36988569b83b7907f92c7a2f4172fca18de90531de"} err="failed to get container status \"eaad778315702b3ca06a8e36988569b83b7907f92c7a2f4172fca18de90531de\": rpc error: code = NotFound desc = could not find container \"eaad778315702b3ca06a8e36988569b83b7907f92c7a2f4172fca18de90531de\": container with ID starting with eaad778315702b3ca06a8e36988569b83b7907f92c7a2f4172fca18de90531de not found: ID does not exist" Nov 04 01:49:01.627965 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.627930 5995 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p"] Nov 04 01:49:01.674850 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:01.674815 5995 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a96a99-openshift-default-6ccb446cf5-sbp2p"] Nov 04 01:49:02.127497 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:02.127472 5995 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6476dc98-8c94-4c02-afb2-c47c54263e00" path="/var/lib/kubelet/pods/6476dc98-8c94-4c02-afb2-c47c54263e00/volumes" Nov 04 01:49:19.642323 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:49:19.642283 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:49:44.167504 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:44.167473 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-37de0964-x92rk_11668fdb-483c-4c4d-a5ee-8c4bbde84649/revoke-keys/0.log" Nov 04 01:49:44.637828 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:44.637798 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-942af0b5-77rgr_2de10d4f-f027-46c7-bfdd-ddf89c34f7dc/revoke-keys/0.log" Nov 04 01:49:47.648062 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:47.648033 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6qnlj_5062a83f-5161-447e-a161-72f1422286d2/manager/0.log" Nov 04 01:49:47.894131 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:47.894103 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pbz49_4e005554-58f5-4e78-9611-043b8d3ccd6a/kuadrant-console-plugin/0.log" Nov 04 01:49:48.680665 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:48.680631 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lzpmm_c282fd58-c58f-4a46-9e63-ab07b0eb2069/manager/0.log" Nov 04 01:49:48.950853 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:48.950826 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jf7b5_d1825e94-4a7d-4ea5-ad61-b1b4fa486373/limitador/0.log" Nov 04 01:49:49.640933 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:49:49.640897 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:49:50.482704 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:50.482672 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-gfwmm_51aec6f1-b80d-498b-b762-892e032f4999/discovery/0.log" Nov 04 01:49:50.736433 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:50.736352 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54cfc47d9b-5tzlw_2c2206b8-9e89-423c-92f7-a7ed330d7421/kube-auth-proxy/0.log" Nov 04 01:49:51.502540 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:51.502496 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-8xfgm_a09c73db-08f2-4a52-8e78-b0073127fa21/istio-proxy/0.log" Nov 04 01:49:51.998131 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:51.998099 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7454465567-qvplm_72c406e7-7705-4f34-8771-31113b11cd86/istio-proxy/0.log" Nov 04 01:49:52.703216 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:52.703186 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f5c95ffcf-hl6wf_f2df8dc5-6403-4827-b6d6-9c209c595e05/router/0.log" Nov 04 01:49:55.306904 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:49:55.306877 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-r46c8_4141f1e2-a242-463e-977b-87ca0ba3b369/main/0.log" Nov 04 01:50:19.641252 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:50:19.641210 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:50:26.118978 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:26.118945 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b2jgm_3ae06262-6e59-4775-bedc-5305ace1fae6/global-pull-secret-syncer/0.log" Nov 04 01:50:27.998887 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:27.998857 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-65nzk_a23c835a-aacb-4f92-a2cd-799f5e870b85/connection-checker/0.log" Nov 04 01:50:28.435068 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:28.434988 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-d26rm_cd605a92-f88c-40ea-bde7-53c2da66a576/connection-checker/0.log" Nov 04 01:50:28.911407 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:28.911374 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xzhqb_a8b2b155-18ef-40a1-b87b-a6e44e130d65/connection-checker/0.log" Nov 04 01:50:29.812081 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:29.812050 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qvjtd_a06edce6-baf3-40bc-9d0e-a6e71c393165/konnectivity-agent/0.log" Nov 04 01:50:31.650386 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:31.650356 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-179.us-west-2.compute.internal_5be03f35a536944b4cf2058a360cdcb6/haproxy/0.log" Nov 04 01:50:36.052309 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.052281 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6qnlj_5062a83f-5161-447e-a161-72f1422286d2/manager/0.log" Nov 04 01:50:36.097922 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.097893 5995 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn"] Nov 04 01:50:36.098297 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.098284 5995 cpu_manager.go:414] "containerMap: removing container" podUID="6476dc98-8c94-4c02-afb2-c47c54263e00" containerName="istio-proxy" Nov 04 01:50:36.098343 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.098298 5995 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6476dc98-8c94-4c02-afb2-c47c54263e00" containerName="istio-proxy" Nov 04 01:50:36.098380 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.098370 5995 memory_manager.go:365] "RemoveStaleState removing state" podUID="6476dc98-8c94-4c02-afb2-c47c54263e00" containerName="istio-proxy" Nov 04 01:50:36.101606 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.101587 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:36.151086 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.151058 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pfwh8\"/\"openshift-service-ca.crt\"" Nov 04 01:50:36.165478 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.165442 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f96a8e6d-6729-41ba-a0fe-688f2e23d6e6-sys\") pod \"perf-node-gather-daemonset-wq5gn\" (UID: \"f96a8e6d-6729-41ba-a0fe-688f2e23d6e6\") " pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:36.165635 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.165484 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f96a8e6d-6729-41ba-a0fe-688f2e23d6e6-podres\") pod \"perf-node-gather-daemonset-wq5gn\" (UID: \"f96a8e6d-6729-41ba-a0fe-688f2e23d6e6\") " pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:36.165635 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.165507 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f96a8e6d-6729-41ba-a0fe-688f2e23d6e6-lib-modules\") pod \"perf-node-gather-daemonset-wq5gn\" (UID: \"f96a8e6d-6729-41ba-a0fe-688f2e23d6e6\") " pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:36.165635 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.165610 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n74fc\" (UniqueName: \"kubernetes.io/projected/f96a8e6d-6729-41ba-a0fe-688f2e23d6e6-kube-api-access-n74fc\") pod \"perf-node-gather-daemonset-wq5gn\" (UID: \"f96a8e6d-6729-41ba-a0fe-688f2e23d6e6\") " pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:36.165770 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.165649 5995 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f96a8e6d-6729-41ba-a0fe-688f2e23d6e6-proc\") pod \"perf-node-gather-daemonset-wq5gn\" (UID: \"f96a8e6d-6729-41ba-a0fe-688f2e23d6e6\") " pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:36.197815 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.197785 5995 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pfwh8\"/\"default-dockercfg-btrqg\"" Nov 04 01:50:36.197815 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.197796 5995 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pfwh8\"/\"kube-root-ca.crt\"" Nov 04 01:50:36.199815 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.199789 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn"] Nov 04 01:50:36.266640 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.266607 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f96a8e6d-6729-41ba-a0fe-688f2e23d6e6-proc\") pod \"perf-node-gather-daemonset-wq5gn\" (UID: \"f96a8e6d-6729-41ba-a0fe-688f2e23d6e6\") " pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:36.266835 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.266695 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f96a8e6d-6729-41ba-a0fe-688f2e23d6e6-sys\") pod \"perf-node-gather-daemonset-wq5gn\" (UID: \"f96a8e6d-6729-41ba-a0fe-688f2e23d6e6\") " pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:36.266835 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.266719 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f96a8e6d-6729-41ba-a0fe-688f2e23d6e6-podres\") pod \"perf-node-gather-daemonset-wq5gn\" (UID: \"f96a8e6d-6729-41ba-a0fe-688f2e23d6e6\") " pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:36.266835 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.266734 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f96a8e6d-6729-41ba-a0fe-688f2e23d6e6-lib-modules\") pod \"perf-node-gather-daemonset-wq5gn\" (UID: \"f96a8e6d-6729-41ba-a0fe-688f2e23d6e6\") " pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:36.266835 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.266738 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f96a8e6d-6729-41ba-a0fe-688f2e23d6e6-proc\") pod \"perf-node-gather-daemonset-wq5gn\" (UID: \"f96a8e6d-6729-41ba-a0fe-688f2e23d6e6\") " pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:36.266835 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.266755 5995 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n74fc\" (UniqueName: \"kubernetes.io/projected/f96a8e6d-6729-41ba-a0fe-688f2e23d6e6-kube-api-access-n74fc\") pod \"perf-node-gather-daemonset-wq5gn\" (UID: \"f96a8e6d-6729-41ba-a0fe-688f2e23d6e6\") " pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:36.266835 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.266822 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f96a8e6d-6729-41ba-a0fe-688f2e23d6e6-podres\") pod \"perf-node-gather-daemonset-wq5gn\" (UID: \"f96a8e6d-6729-41ba-a0fe-688f2e23d6e6\") " pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:36.267067 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.266861 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f96a8e6d-6729-41ba-a0fe-688f2e23d6e6-sys\") pod \"perf-node-gather-daemonset-wq5gn\" (UID: \"f96a8e6d-6729-41ba-a0fe-688f2e23d6e6\") " pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:36.267067 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.266867 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f96a8e6d-6729-41ba-a0fe-688f2e23d6e6-lib-modules\") pod \"perf-node-gather-daemonset-wq5gn\" (UID: \"f96a8e6d-6729-41ba-a0fe-688f2e23d6e6\") " pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:36.327100 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.327026 5995 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n74fc\" (UniqueName: \"kubernetes.io/projected/f96a8e6d-6729-41ba-a0fe-688f2e23d6e6-kube-api-access-n74fc\") pod \"perf-node-gather-daemonset-wq5gn\" (UID: \"f96a8e6d-6729-41ba-a0fe-688f2e23d6e6\") " pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:36.413927 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.413889 5995 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:36.489197 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.489167 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pbz49_4e005554-58f5-4e78-9611-043b8d3ccd6a/kuadrant-console-plugin/0.log" Nov 04 01:50:36.785681 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.785647 5995 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn"] Nov 04 01:50:36.792397 ip-10-0-138-179 kubenswrapper[5995]: W1104 01:50:36.792368 5995 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf96a8e6d_6729_41ba_a0fe_688f2e23d6e6.slice/crio-4cd0d09f21163feb6d4b535d8b0983cd23fb515509872561e8df70a0ac14d5e2 WatchSource:0}: Error finding container 4cd0d09f21163feb6d4b535d8b0983cd23fb515509872561e8df70a0ac14d5e2: Status 404 returned error can't find the container with id 4cd0d09f21163feb6d4b535d8b0983cd23fb515509872561e8df70a0ac14d5e2 Nov 04 01:50:36.968735 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.968697 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" event={"ID":"f96a8e6d-6729-41ba-a0fe-688f2e23d6e6","Type":"ContainerStarted","Data":"58ec4e47024bbd623df0f966837d8d44c030dd6e6f0269c961b7fbbe3100c8e4"} Nov 04 01:50:36.968904 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.968757 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:36.968904 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:36.968773 5995 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" event={"ID":"f96a8e6d-6729-41ba-a0fe-688f2e23d6e6","Type":"ContainerStarted","Data":"4cd0d09f21163feb6d4b535d8b0983cd23fb515509872561e8df70a0ac14d5e2"} Nov 04 01:50:37.082167 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:37.082054 5995 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" podStartSLOduration=1.082039846 podStartE2EDuration="1.082039846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:50:37.08097332 +0000 UTC m=+2387.112706709" watchObservedRunningTime="2026-11-04 01:50:37.082039846 +0000 UTC m=+2387.113773296" Nov 04 01:50:37.331879 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:37.331841 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lzpmm_c282fd58-c58f-4a46-9e63-ab07b0eb2069/manager/0.log" Nov 04 01:50:37.678978 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:37.678944 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jf7b5_d1825e94-4a7d-4ea5-ad61-b1b4fa486373/limitador/0.log" Nov 04 01:50:42.984291 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:42.984262 5995 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pfwh8/perf-node-gather-daemonset-wq5gn" Nov 04 01:50:43.927728 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:43.927693 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-fh4kl_75c5b017-8f43-41dc-b79b-0353de290289/cluster-monitoring-operator/0.log" Nov 04 01:50:44.002537 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:44.002507 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-gfwmm_51aec6f1-b80d-498b-b762-892e032f4999/discovery/0.log" Nov 04 01:50:44.108235 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:44.108206 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54cfc47d9b-5tzlw_2c2206b8-9e89-423c-92f7-a7ed330d7421/kube-auth-proxy/0.log" Nov 04 01:50:44.735280 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:44.735248 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-8xfgm_a09c73db-08f2-4a52-8e78-b0073127fa21/istio-proxy/0.log" Nov 04 01:50:45.164864 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:45.164784 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7454465567-qvplm_72c406e7-7705-4f34-8771-31113b11cd86/istio-proxy/0.log" Nov 04 01:50:45.795490 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:45.795459 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f5c95ffcf-hl6wf_f2df8dc5-6403-4827-b6d6-9c209c595e05/router/0.log" Nov 04 01:50:46.553510 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:46.553479 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kk42p_820aaae5-2a74-40c6-a009-61edcd7cdfe5/node-exporter/0.log" Nov 04 01:50:46.991643 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:46.991615 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kk42p_820aaae5-2a74-40c6-a009-61edcd7cdfe5/kube-rbac-proxy/0.log" Nov 04 01:50:47.426092 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:47.426011 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kk42p_820aaae5-2a74-40c6-a009-61edcd7cdfe5/init-textfile/0.log" Nov 04 01:50:49.641998 ip-10-0-138-179 kubenswrapper[5995]: E1104 01:50:49.641965 5995 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4j876" containerName="extractor" Nov 04 01:50:49.878116 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:49.878083 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6qnlj_5062a83f-5161-447e-a161-72f1422286d2/manager/0.log" Nov 04 01:50:50.094025 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:50.093995 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pbz49_4e005554-58f5-4e78-9611-043b8d3ccd6a/kuadrant-console-plugin/0.log" Nov 04 01:50:50.425291 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:50.425217 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f_5600ddae-56c7-4c93-afff-2900207b0819/extract/0.log" Nov 04 01:50:50.431139 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:50.431105 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644jws8f_5600ddae-56c7-4c93-afff-2900207b0819/extract/0.log" Nov 04 01:50:50.435160 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:50.435135 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:50:50.441076 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:50.441041 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8cfg7_ae64edd1-43dc-406d-bcad-f7f34bb1f991/console-operator/1.log" Nov 04 01:50:50.694136 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:50.694105 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lzpmm_c282fd58-c58f-4a46-9e63-ab07b0eb2069/manager/0.log" Nov 04 01:50:50.808640 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:50.808609 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jf7b5_d1825e94-4a7d-4ea5-ad61-b1b4fa486373/limitador/0.log" Nov 04 01:50:56.062045 ip-10-0-138-179 kubenswrapper[5995]: I1104 01:50:56.062016 5995 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-47hlt_747ed7ed-d05b-434a-8360-acc2292c2257/prometheus-operator-admission-webhook/0.log"